Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Michal Hocko <mhocko@suse.com>, Peter Xu <peterx@redhat.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        James Houghton <jthoughton@google.com>,
        Mina Almasry <almasrymina@google.com>,
        "Aneesh Kumar K . V" <aneesh.kumar@linux.vnet.ibm.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Christian Borntraeger <borntraeger@linux.ibm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [RFC PATCH 2/3] hugetlb: do not update address in huge_pmd_unshare
Date:   Fri, 27 May 2022 15:58:48 -0700
Message-Id: <20220527225849.284839-3-mike.kravetz@oracle.com>
In-Reply-To: <20220527225849.284839-1-mike.kravetz@oracle.com>
References: <20220527225849.284839-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?vk1bUjEFIz+LdaSqiBOJajAqpl9sb+zg2w0mjtEwmKbVlZ9eI+2vVW0jQXN5?=
 =?us-ascii?Q?0+r2ephpO3AMAFhFl5m39XVjtnK20eoI0bU+JQw96AKrFrrZnAJp7YInuxqO?=
 =?us-ascii?Q?EQZeo8cUIpoFs8Z5YZVBepjHpYqosnsYG7UyutoxJUkJBj/Iwmf2gsLnNF8y?=
 =?us-ascii?Q?RYIr9XY0AuHLSjOwqIJG/aNj0EEYwcEChwH3JNTtjeFldBXa18ogq9v3tMMI?=
 =?us-ascii?Q?3Fl9R7vITSXNU6ixDr3eeh36a5c5mzEnqOtNIbEJBsoqdZj/6uE56dRkKXU0?=
 =?us-ascii?Q?N9H/iUw+KbjJ1cOCLxkOpYJPnRLe054oVN3OvF6qG6xXKerFWQxOAh+lk/ae?=
 =?us-ascii?Q?XumowqhyNqidQxO69J8vI/XROhcp1UsaGlH3bKNulAwImKSfBcwk0MfeySSh?=
 =?us-ascii?Q?N+yEHm+0x4FOhmdir7b6FqEBG3Gn6SV/OV48z6+Y8KDqUkbu0xu3OlQWM3s9?=
 =?us-ascii?Q?klviMoS6nQGjXrO/gcR9KEHnBIK+ChyZ/URDh1Oa8hmCEylAw9nnBctuUVWc?=
 =?us-ascii?Q?8QoQc1hIpuqF8fE+PKQ+HF84pd1byylzky1kjynBZsaEtqazl+6WufY6p3RU?=
 =?us-ascii?Q?fbCewjzm6TLssgI9Mx4/F1gtVPSKQhK/rhpWExV/xeR3XdWW/Javp6nJ3j7s?=
 =?us-ascii?Q?YJaSdBRzx/SqlpbMFCHMet3BuLVthmzN9twfWgC3Icq6BAvcqXPp2vReUchb?=
 =?us-ascii?Q?JX6YPuBKMHOQgUsuE8Up4JIXhoXuzTcVmtxwGj7l/mhkjZCT0rVjMID5chjE?=
 =?us-ascii?Q?9bt1J+WWqVO2J08SzarpXJGs/Ff2tblTEfVyS6d7GwGkr5H7/icSKQrOdbcw?=
 =?us-ascii?Q?lfrOt+V6AcSUaVcmMscZ2IsDgXCdL6kolxXoSyKDyqKuLMzMNL46PMKFjGYw?=
 =?us-ascii?Q?Er/C/+jMCbUHmc5wn+pjhQKLhTU1ZrYV9dQp+dnpzHak5Io0pqxBdx5T1nrZ?=
 =?us-ascii?Q?Fa1J3kVqpbpvIIpm7oIcvS42lu2uOaWIacQmf76xx5Ox/jqhZgd1JxoOXU4O?=
 =?us-ascii?Q?R9+2PJbqPeYzB7Li7DCWl/u1VnlkiHVQl37CuLsAr4wWDHU3c4BT34QX9k/W?=
 =?us-ascii?Q?T0PbcUgf234FZ2rEQRyLafEPOIhCfPQ8IymYpOeVNgEjn8wzW66OPlZjFb4E?=
 =?us-ascii?Q?y1+5kEX/MsP+NkALANpr56YzE3C89d2gLpcTYWVUDLJQ6O0Nk/4PGfrvGFFb?=
 =?us-ascii?Q?e4U1RDARoPd1juqjP0J+kBPmm1SMIfSRoMNRvLOqeqmGo+NxSyxKE4pN349W?=
 =?us-ascii?Q?sm47276n0Qq4eNPeE/oZNfrF6iHAlvOpQaoe/3TsCIFvv6TZISLudTDFP02y?=
 =?us-ascii?Q?13OeW51CH7ctu+Ab376BFEd4ee3nKJ7W+cxUMYTBZV7PNnMdSwvTMVefkDXk?=
 =?us-ascii?Q?p70t48rZbCBiUC+69HparD/nGtvf7o/Mtga24dDjHwrxVS9ltqkHOYcROQM+?=
 =?us-ascii?Q?BThYCRPc1Q+m7TMzBNYMawmWbZNKhpU+w/u7vlum4WSO/sVsL2INCei2pB5u?=
 =?us-ascii?Q?FQ9FOfqPo9SFKTNT693UUahEGt0JbUIC4uqPC2rKSrX8E6J51HAJN/ICsh3j?=
 =?us-ascii?Q?AEKp3zGGXc81IoX9CIki1NGu4QNv4QLsagA8vyljpfN60M6jd8UqCwhh0Iur?=
 =?us-ascii?Q?yEFG8Jla66gClvdH7NULlFSKjan47qM9dDhtxFLrGWX+3yljgVihdg9KpzTa?=
 =?us-ascii?Q?dixvamRr6JxbpwEbuBuCC7O7+Vi96xgWdwOmDkxqsCKU95T5FHTbLjlD3nVS?=
 =?us-ascii?Q?mKNVGmnVMOyLAziZ/eSZ+QeY+vDYn+U=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 673bd517-9cf6-44da-b946-08da40347f64
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2022 22:59:04.8474
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: bgB9tGkDMie7KxZrrHHo84Gg7pwqqqAITR8rg1tLN3uv0r+o7sWYHHOy4xo5JU2+3TcVvk7uzvh+hLQaaVPwYA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB5068
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.874
 definitions=2022-05-27_07:2022-05-27,2022-05-27 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0
 malwarescore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2205270111
X-Proofpoint-GUID: ak9l4zlU8484jy7Y2cVramGWqttby6aF
X-Proofpoint-ORIG-GUID: ak9l4zlU8484jy7Y2cVramGWqttby6aF
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

As an optimization for loops sequentially processing hugetlb address
ranges, huge_pmd_unshare would update a passed address if it unshared a
pmd.  Updating a loop control variable outside the loop like this is
generally a bad idea.  These loops are now using hugetlb_mask_last_hp
to optimize scanning when non-present ptes are discovered.  The same
can be done when huge_pmd_unshare returns 1 indicating a pmd was
unshared.

Remove address update from huge_pmd_unshare.  Change the passed argument
type and update all callers.  In loops sequentially processing addresses
use hugetlb_mask_last_hp to update address if pmd is unshared.

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
 include/linux/hugetlb.h |  4 ++--
 mm/hugetlb.c            | 46 ++++++++++++++++++-----------------------
 mm/rmap.c               |  4 ++--
 3 files changed, 24 insertions(+), 30 deletions(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 25078a0ea1d8..307c8f6e6752 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -196,7 +196,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
 unsigned long hugetlb_mask_last_hp(struct hstate *h);
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-				unsigned long *addr, pte_t *ptep);
+				unsigned long addr, pte_t *ptep);
 void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
 				unsigned long *start, unsigned long *end);
 struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
@@ -243,7 +243,7 @@ static inline struct address_space *hugetlb_page_mapping_lock_write(
 
 static inline int huge_pmd_unshare(struct mm_struct *mm,
 					struct vm_area_struct *vma,
-					unsigned long *addr, pte_t *ptep)
+					unsigned long addr, pte_t *ptep)
 {
 	return 0;
 }
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a2db878b2255..c7d3fbf3ec05 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4940,7 +4940,6 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long old_end = old_addr + len;
 	unsigned long last_addr_mask;
-	unsigned long old_addr_copy;
 	pte_t *src_pte, *dst_pte;
 	struct mmu_notifier_range range;
 	bool shared_pmd = false;
@@ -4968,14 +4967,10 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
 		if (huge_pte_none(huge_ptep_get(src_pte)))
 			continue;
 
-		/* old_addr arg to huge_pmd_unshare() is a pointer and so the
-		 * arg may be modified. Pass a copy instead to preserve the
-		 * value in old_addr.
-		 */
-		old_addr_copy = old_addr;
-
-		if (huge_pmd_unshare(mm, vma, &old_addr_copy, src_pte)) {
+		if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) {
 			shared_pmd = true;
+			old_addr |= last_addr_mask;
+			new_addr |= last_addr_mask;
 			continue;
 		}
 
@@ -5040,10 +5035,11 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct
 		}
 
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+		if (huge_pmd_unshare(mm, vma, address, ptep)) {
 			spin_unlock(ptl);
 			tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
 			force_flush = true;
+			address |= last_addr_mask;
 			continue;
 		}
 
@@ -6327,7 +6323,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 			continue;
 		}
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+		if (huge_pmd_unshare(mm, vma, address, ptep)) {
 			/*
 			 * When uffd-wp is enabled on the vma, unshare
 			 * shouldn't happen at all.  Warn about it if it
@@ -6337,6 +6333,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 			pages++;
 			spin_unlock(ptl);
 			shared_pmd = true;
+			address |= last_addr_mask;
 			continue;
 		}
 		pte = huge_ptep_get(ptep);
@@ -6760,11 +6757,11 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
  *	    0 the underlying pte page is not shared, or it is the last user
  */
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-					unsigned long *addr, pte_t *ptep)
+					unsigned long addr, pte_t *ptep)
 {
-	pgd_t *pgd = pgd_offset(mm, *addr);
-	p4d_t *p4d = p4d_offset(pgd, *addr);
-	pud_t *pud = pud_offset(p4d, *addr);
+	pgd_t *pgd = pgd_offset(mm, addr);
+	p4d_t *p4d = p4d_offset(pgd, addr);
+	pud_t *pud = pud_offset(p4d, addr);
 
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
 	BUG_ON(page_count(virt_to_page(ptep)) == 0);
@@ -6774,14 +6771,6 @@ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 	pud_clear(pud);
 	put_page(virt_to_page(ptep));
 	mm_dec_nr_pmds(mm);
-	/*
-	 * This update of passed address optimizes loops sequentially
-	 * processing addresses in increments of huge page size (PMD_SIZE
-	 * in this case).  By clearing the pud, a PUD_SIZE area is unmapped.
-	 * Update address to the 'last page' in the cleared area so that
-	 * calling loop can move to first page past this area.
-	 */
-	*addr |= PUD_SIZE - PMD_SIZE;
 	return 1;
 }
 
@@ -6793,7 +6782,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 }
 
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-				unsigned long *addr, pte_t *ptep)
+				unsigned long addr, pte_t *ptep)
 {
 	return 0;
 }
@@ -6902,6 +6891,13 @@ unsigned long hugetlb_mask_last_hp(struct hstate *h)
 /* See description above.  Architectures can provide their own version. */
 __weak unsigned long hugetlb_mask_last_hp(struct hstate *h)
 {
+	unsigned long hp_size = huge_page_size(h);
+
+#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
+	if (hp_size == PMD_SIZE)	/* required for pmd sharing */
+		return PUD_SIZE - PMD_SIZE;
+#endif
+
 	return ~(0);
 }
 
@@ -7128,14 +7124,12 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
 	mmu_notifier_invalidate_range_start(&range);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
 	for (address = start; address < end; address += PUD_SIZE) {
-		unsigned long tmp = address;
-
 		ptep = huge_pte_offset(mm, address, sz);
 		if (!ptep)
 			continue;
 		ptl = huge_pte_lock(h, mm, ptep);
 		/* We don't want 'address' to be changed */
-		huge_pmd_unshare(mm, vma, &tmp, ptep);
+		huge_pmd_unshare(mm, vma, address, ptep);
 		spin_unlock(ptl);
 	}
 	flush_hugetlb_tlb_range(vma, start, end);
diff --git a/mm/rmap.c b/mm/rmap.c
index 5bcb334cd6f2..45b04e2e83ab 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1559,7 +1559,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				 */
 				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
 
-				if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) {
+				if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
 					flush_tlb_range(vma, range.start, range.end);
 					mmu_notifier_invalidate_range(mm, range.start,
 								      range.end);
@@ -1923,7 +1923,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 				 */
 				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
 
-				if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) {
+				if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
 					flush_tlb_range(vma, range.start, range.end);
 					mmu_notifier_invalidate_range(mm, range.start,
 								      range.end);
-- 
2.35.3