Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Michal Hocko <mhocko@suse.com>, Peter Xu <peterx@redhat.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        James Houghton <jthoughton@google.com>,
        Mina Almasry <almasrymina@google.com>,
        "Aneesh Kumar K . V" <aneesh.kumar@linux.vnet.ibm.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Christian Borntraeger <borntraeger@linux.ibm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [RFC PATCH 1/3] hugetlb: skip to end of PT page mapping when pte not present
Date:   Fri, 27 May 2022 15:58:47 -0700
Message-Id: <20220527225849.284839-2-mike.kravetz@oracle.com>
In-Reply-To: <20220527225849.284839-1-mike.kravetz@oracle.com>
References: <20220527225849.284839-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2lDWOWtqURVwYl4lbdgQr8szMg1ITLFvfAQ2psuUack8Z85Smj3B+Kz4Wu6G?=
 =?us-ascii?Q?Tdsr3gujDhyg+IFzCO3qaZ8gDU5qGhblekF/DDClP3+xduK9LCdJodKsdo6Y?=
 =?us-ascii?Q?ERwzAWFiwiX/K4kkxVFl4XsK/BId31IMYNM65yH7tweniVM7+N/f67LQ3u7y?=
 =?us-ascii?Q?QkZmijOArvH1mrm34XCOPlhH1v55hu4hnBYEH+fWyGXrV7tqKhrzbGFnRXam?=
 =?us-ascii?Q?hws2gVpqBrdkUk7OfZ30pAREhhAQHLEwRlklByTJ6T20wBcoYMQTjbk7bHoV?=
 =?us-ascii?Q?SKjRf0cV0j4VQHDcIXdIFz2+gMM1cQRW6nJMwU2vcPlVDVwLbClIxlpveb7M?=
 =?us-ascii?Q?h7ngPTG4qWNTEitHGvA/m4NOXZrAIGXksCkPllj0Cdn2OZv0oA41r2iRuDVw?=
 =?us-ascii?Q?woNP17FiI1D/pSYvYy8aw/1H7IV6UESXqyWhZwSHXfPA6bc5jEGQmdSfJ6J4?=
 =?us-ascii?Q?vwJHw8qJGpHtdEfYMXc3V2FtEkut76emWIdyEjfdn+PaBxfQV19YkL0w5wG6?=
 =?us-ascii?Q?Kc184oJV5opV4gmvAJzDKViSycZxSPuPgTfIRwepG+ZZktz+9YAvXWuC61LI?=
 =?us-ascii?Q?W0hjhQ62eXjmMGhV+bRIvCUeeg7nYOwRSAQTkz8thRiI1Vysb249eGA/OPvB?=
 =?us-ascii?Q?e+thCdGA97flMWWXmy3MGEsBiljbvkQHF0vugf5Srtczvnxxpvl7D+jZLMl8?=
 =?us-ascii?Q?2AigNRXOm88/pvluLV1Tu307OagO91brO56lEWzLjcyjKTfAAKtmCO3sU7g3?=
 =?us-ascii?Q?vQ7bWTh0KXxLxFQEeYzLfT5ZNQS77PekH354PqvxCqBiHF8AAIrZYuADr/MJ?=
 =?us-ascii?Q?M2wEPMo8wdg9SKdk0/45pcN4iATk3gdfn2LHqJOaWp4EiQEb73XLHHf7VoYg?=
 =?us-ascii?Q?oEM0sV631E/h0qYMdfuIwRwPHBytO16EbzcnJZalGorIYiFXBOiLqyjMknfo?=
 =?us-ascii?Q?JUzEnRg6D7r0x4IwwnaJ+zCR12gFKq7MpKXtLq+LfU6rnCkm7CkGChQgBqJY?=
 =?us-ascii?Q?PgIEWz/TlxMv0ZWp8jO98NMWZkGGaDxPGEpvySisJ8V4YidlpRvmG9rWfo2a?=
 =?us-ascii?Q?KFGLat3GEK+NzFnTYLEH0e+ozb3uwwf6kSHxjf97mx6sdEOkgxrQbYTYp6k+?=
 =?us-ascii?Q?Ph1CVnROzcyx/Kuh30nDVtyvd7ZPKpgVmoTvoO1ESwnTFbO1lcpl6p/ODRwO?=
 =?us-ascii?Q?3ZfI6b3Uc/9vZwDeqQ6HgTu7jbfmZNvLSBLPADIAbNStccv5Mz9A1/o7Oaw3?=
 =?us-ascii?Q?MzQUG+HnWE5Fu3oAaDpHRRrLHbigrZ8voSDZIwNobufYJ/24he4B8LTk797v?=
 =?us-ascii?Q?lrBZk4spPOOB9mPgS2hZskCBnHGMdtrLIpxplye+qGKqfbX5Zv05O/Q4POVU?=
 =?us-ascii?Q?saX83zeDVNDLeRC4wheEKFgqm211VYx7WX5LXQuqIzSRNzySvwcwWIFrQyub?=
 =?us-ascii?Q?e0nOoGbAFiCg/tc6OwwvKmOkkFGwuz4iNrQzVy1qXOoQEyR585/UeIMnBfql?=
 =?us-ascii?Q?E61CAxnud6zZtbjq062nTn8XgAN7DT2gvQpvtw/GvHCFYzxDarcqcLDxpptI?=
 =?us-ascii?Q?J1k/n1bDYj7Sfui+bYQgJCItrj3g0+x/IqDVxDu9Sxx6V34v/zh+uTtT078B?=
 =?us-ascii?Q?i6RyYx824nv7c7VV8icfVR4y0+YnytKT4PeccaOUUYLutVaCGaRhAkRfvJ56?=
 =?us-ascii?Q?3iDHmuclXor5AVgRgzw71XwFZhcXlG4gAjKOAj4rKCu+CC8osPh1LIS0yb4a?=
 =?us-ascii?Q?Z/7iyZacICXi3Dn8DruMJ9tPaxfPnmU=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c51018e1-f5bf-4799-7b85-08da40347e65
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2022 22:59:03.0652
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kpzUJwew93OyEjoQzxSxSeIwZ2I5bxBuvlgJNeWiJM13UGaY5nlNWRCFfwZUM2Jf4MrGD20s/SkXpxtZZWo6kg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB4831
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.874
 definitions=2022-05-27_07:2022-05-27,2022-05-27 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0
 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2205270111
X-Proofpoint-ORIG-GUID: gYnZrLLda-e_fdbSDCoMVcYP8red8kdU
X-Proofpoint-GUID: gYnZrLLda-e_fdbSDCoMVcYP8red8kdU
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

HugeTLB address ranges are linearly scanned during fork, unmap and
remap operations.  If a non-present entry is encountered, the code
currently continues to the next huge page aligned address.  However,
a non-present entry implies that the page table page for that entry
is not present.  Therefore, the linear scan can skip to the end of
range mapped by the page table page.  This can speed operations on
large sparsely populated hugetlb mappings.

Create a new routine hugetlb_mask_last_hp() that will return an
address mask.  When the mask is ORed with an address, the result
will be the address of the last huge page mapped by the associated
page table page.  Use this mask to update addresses in routines which
linearly scan hugetlb address ranges when a non-present pte is
encountered.

hugetlb_mask_last_hp is related to the implementation of huge_pte_offset
as hugetlb_mask_last_hp is called when huge_pte_offset returns NULL.
This patch only provides a complete hugetlb_mask_last_hp implementation
when CONFIG_ARCH_WANT_GENERAL_HUGETLB is defined.  Architectures which
provide their own versions of huge_pte_offset can also provide their own
version of hugetlb_mask_last_hp.

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
 include/linux/hugetlb.h |  1 +
 mm/hugetlb.c            | 58 +++++++++++++++++++++++++++++++++++++----
 2 files changed, 54 insertions(+), 5 deletions(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index e4cff27d1198..25078a0ea1d8 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -194,6 +194,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz);
 pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
+unsigned long hugetlb_mask_last_hp(struct hstate *h);
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 				unsigned long *addr, pte_t *ptep);
 void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 7c468ac1d069..a2db878b2255 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4731,6 +4731,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 	unsigned long npages = pages_per_huge_page(h);
 	struct address_space *mapping = src_vma->vm_file->f_mapping;
 	struct mmu_notifier_range range;
+	unsigned long last_addr_mask;
 	int ret = 0;
 
 	if (cow) {
@@ -4750,11 +4751,14 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 		i_mmap_lock_read(mapping);
 	}
 
+	last_addr_mask = hugetlb_mask_last_hp(h);
 	for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) {
 		spinlock_t *src_ptl, *dst_ptl;
 		src_pte = huge_pte_offset(src, addr, sz);
-		if (!src_pte)
+		if (!src_pte) {
+			addr |= last_addr_mask;
 			continue;
+		}
 		dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz);
 		if (!dst_pte) {
 			ret = -ENOMEM;
@@ -4771,8 +4775,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 		 * after taking the lock below.
 		 */
 		dst_entry = huge_ptep_get(dst_pte);
-		if ((dst_pte == src_pte) || !huge_pte_none(dst_entry))
+		if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) {
+			addr |= last_addr_mask;
 			continue;
+		}
 
 		dst_ptl = huge_pte_lock(h, dst, dst_pte);
 		src_ptl = huge_pte_lockptr(h, src, src_pte);
@@ -4933,6 +4939,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
 	unsigned long sz = huge_page_size(h);
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long old_end = old_addr + len;
+	unsigned long last_addr_mask;
 	unsigned long old_addr_copy;
 	pte_t *src_pte, *dst_pte;
 	struct mmu_notifier_range range;
@@ -4948,12 +4955,16 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
 	flush_cache_range(vma, range.start, range.end);
 
 	mmu_notifier_invalidate_range_start(&range);
+	last_addr_mask = hugetlb_mask_last_hp(h);
 	/* Prevent race with file truncation */
 	i_mmap_lock_write(mapping);
 	for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
 		src_pte = huge_pte_offset(mm, old_addr, sz);
-		if (!src_pte)
+		if (!src_pte) {
+			old_addr |= last_addr_mask;
+			new_addr |= last_addr_mask;
 			continue;
+		}
 		if (huge_pte_none(huge_ptep_get(src_pte)))
 			continue;
 
@@ -4998,6 +5009,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct
 	struct hstate *h = hstate_vma(vma);
 	unsigned long sz = huge_page_size(h);
 	struct mmu_notifier_range range;
+	unsigned long last_addr_mask;
 	bool force_flush = false;
 
 	WARN_ON(!is_vm_hugetlb_page(vma));
@@ -5018,11 +5030,14 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct
 				end);
 	adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
 	mmu_notifier_invalidate_range_start(&range);
+	last_addr_mask = hugetlb_mask_last_hp(h);
 	address = start;
 	for (; address < end; address += sz) {
 		ptep = huge_pte_offset(mm, address, sz);
-		if (!ptep)
+		if (!ptep) {
+			address |= last_addr_mask;
 			continue;
+		}
 
 		ptl = huge_pte_lock(h, mm, ptep);
 		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
@@ -6285,6 +6300,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 	unsigned long pages = 0, psize = huge_page_size(h);
 	bool shared_pmd = false;
 	struct mmu_notifier_range range;
+	unsigned long last_addr_mask;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 
@@ -6301,12 +6317,15 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 	flush_cache_range(vma, range.start, range.end);
 
 	mmu_notifier_invalidate_range_start(&range);
+	last_addr_mask = hugetlb_mask_last_hp(h);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
 	for (; address < end; address += psize) {
 		spinlock_t *ptl;
 		ptep = huge_pte_offset(mm, address, psize);
-		if (!ptep)
+		if (!ptep) {
+			address |= last_addr_mask;
 			continue;
+		}
 		ptl = huge_pte_lock(h, mm, ptep);
 		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
 			/*
@@ -6857,6 +6876,35 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 	return (pte_t *)pmd;
 }
 
+/*
+ * Return a mask that can be used to update an address to the last huge
+ * page in a page table page mapping size.  Used to skip non-present
+ * page table entries when linearly scanning address ranges.  Architectures
+ * with unique huge page to page table relationships can define their own
+ * version of this routine.
+ */
+unsigned long hugetlb_mask_last_hp(struct hstate *h)
+{
+	unsigned long hp_size = huge_page_size(h);
+
+	if (hp_size == P4D_SIZE)
+		return PGDIR_SIZE - P4D_SIZE;
+	else if (hp_size == PUD_SIZE)
+		return P4D_SIZE - PUD_SIZE;
+	else if (hp_size == PMD_SIZE)
+		return PUD_SIZE - PMD_SIZE;
+
+	return ~(0);
+}
+
+#else
+
+/* See description above.  Architectures can provide their own version. */
+__weak unsigned long hugetlb_mask_last_hp(struct hstate *h)
+{
+	return ~(0);
+}
+
 #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */
 
 /*
-- 
2.35.3