Received-SPF: pass (google.com: domain of linux-kernel+bounces-341-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <941c2269-98f0-43a7-a516-cb5b4909ae7f@intel.com>
Date: Fri, 15 Dec 2023 10:26:32 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v1 14/39] mm/rmap: introduce
 folio_add_anon_rmap_[pte|ptes|pmd]()
Content-Language: en-US
To: David Hildenbrand <david@redhat.com>, <linux-kernel@vger.kernel.org>
CC: <linux-mm@kvack.org>, Andrew Morton <akpm@linux-foundation.org>, "Matthew
 Wilcox (Oracle)" <willy@infradead.org>, Hugh Dickins <hughd@google.com>, Ryan
 Roberts <ryan.roberts@arm.com>, Mike Kravetz <mike.kravetz@oracle.com>,
	Muchun Song <muchun.song@linux.dev>, Peter Xu <peterx@redhat.com>
References: <20231211155652.131054-1-david@redhat.com>
 <20231211155652.131054-15-david@redhat.com>
From: "Yin, Fengwei" <fengwei.yin@intel.com>
In-Reply-To: <20231211155652.131054-15-david@redhat.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QWxnMWdEVTQyNS9PL3JjN3UvU3J6MENuQWh2QTNzS2pqT25qZC9pOU9oYWVB?=
 =?utf-8?B?L2Z0VC9jcTFBbC9MVDBHSHhSbGJNWTR2S1FQZjVpclVFQXhkZ2ZKNWxWdzJ4?=
 =?utf-8?B?WXBhY0lQYXRJbHhVT2RNdis3TEJzaXhrejVMTm0vTGtpU01NSGpVOWlLUDY5?=
 =?utf-8?B?VWRUbWJZcHlSdTZITlhtVzFTcTJUZ3hOVmxMZWRsU0FUU0FET09adWZOSEpu?=
 =?utf-8?B?d0FENXNydHl5ZnFXTGV4N0xRS0VTTFRTOEV4anRTUWNLOXZWWjFjclE0MHV3?=
 =?utf-8?B?anFTUW1YTEZsSFEvMEVxbUluMEpYUHkxcWJ1SlZEa0JjdktEZU9oQVJTVStZ?=
 =?utf-8?B?eUtxLzBHcjNPNzhkK1FsTnJLK0xtc3lFbTJiQ0FGZDZwMjFzYTg2ZDBUaG5R?=
 =?utf-8?B?UnB4YzZVbVJKd1BBRFV5ckQrcDRHU0h2OU02MEFVWWpMTG0rR1BLanpwOUJu?=
 =?utf-8?B?ZVF5WHhMVHdCd1JUUE1MVmZNWnR1TmJpalNESE15ckZCMElFK3A0a2xMWHNw?=
 =?utf-8?B?SVN4OGw0d1RKeDZBZHZPWS9tSlM1ZUNzd2NjQnQ2MllWYXBHeGc4OG1saHVF?=
 =?utf-8?B?UVBRU0M0Q1diRTEwaTRCaFRHYmhqU1VjWXpYMGhSNEd1ZXR3QVdIYi9TUDJM?=
 =?utf-8?B?SHRmWWxNcVROY1ZBY3FLRlEwRWEzaTAxbUxDaVZpWjNhN3RFV0MwbzBwczhj?=
 =?utf-8?B?T091Q3Y4T2FtRVlpQVVlWS9oeVZxWWxGNDlINFRINjJETS9pVjZiOHFOdWRZ?=
 =?utf-8?B?ZnFrblVZTWFibzJsc09IWHl1MDhNZHNyVmFqZ0YzNERFUVNjNllROS9UbnU1?=
 =?utf-8?B?TFcxODJGUWt2aDNLYjJvY1NVL0hFT2NzSkgvVURQRFk5M2xkcmFpWXJWTEZ1?=
 =?utf-8?B?M0JiRldLVkh2NVNNV25xZUR6UEhPbU9JR3BZdlVsMCsyL1NFRmdJQWRHb2h6?=
 =?utf-8?B?Y24vWld4UkNwWlY4NUlnVEdnT3pHeG5PNTVLVnpQdHVVQ2hkN0Z1WEFOVkdE?=
 =?utf-8?B?YVdJVWYwOXBuemRhS1B0RGNSOVgwWE5xdVZ4bHZRVXl6bVRZUmx6QVNhTHlq?=
 =?utf-8?B?eUlmZks0RU5kUTAxMkg4ZWI3azJldk1WeEJmRGVsQWJrSWZGRWNYYmF3NjZQ?=
 =?utf-8?B?OVdCVWlML0l0cWRoWUE4UE5KT2J0azFnTjRMN1dzOWxWUTFlR3YyRFNod0to?=
 =?utf-8?B?RW54dE1MVW12V0JuQ0xKTXlNTVVqOHNpakduVVhHVU0yQ1ZHbGpzZG1FVXds?=
 =?utf-8?B?Ukh6UFljSWQwN3AraEFlWUNSSUE0bjZxKzJQanY3TVB2ZmFYSE5Hb3NPRU9U?=
 =?utf-8?B?RzJ5Rm1QVkJpSjk3YnhubFpXMVNPOXV5UklCSDgzOU1rYm81ZDI2WjVlZVpR?=
 =?utf-8?B?TkZtUnJCNEZLUTkzQmMzcm1pVHYwdGl3ejN2WmpYQWFoYjRNVXdEalFOaHdi?=
 =?utf-8?B?TnpSRHNMYW9ZcE5CS21ia2d3ZzRUTVUxZVY3S3BkMXlCbFFBNHRUSWtrMGNC?=
 =?utf-8?B?bU9WVTUxbG9HMjV6eHpjeHB5cGt5a2RDYmF2bEJPZFJxUHMrYjREZXIwYnFa?=
 =?utf-8?B?aTFJL1RXRHBLZDBpbUlhbnEvdkxSM3BBSzlyZUErS3JzbU1ZRXEwOXh0OEVw?=
 =?utf-8?B?ck01aFRaMTE5dFhCamM5TmEvZGwrWUpCa2JzL1MwOGtJWHF5REJSQTN2UGZW?=
 =?utf-8?B?MHlhdlkzRXpjUlJXZ0FITldzUVVBcE50VDBudU0vS3QxQmlMTVlObFlHdG1p?=
 =?utf-8?B?VGlLNE43Z3cyMmhhNmk2b3dYaG9yNGZKOU1rSER5SVRaOGh5QVVaRjdsSHFU?=
 =?utf-8?B?Y0ZGK1Rxd0NhTDQ4bDhVdzNEenNlazF5TEtkNk5UcVQvQVZHcXorZzJaY2JU?=
 =?utf-8?B?Tjl1T3JObXJrNmFaeHpqN3RQemo1WmdvZ2pqZ043OTVhdkV6OTFFYUVqbnFr?=
 =?utf-8?B?OVBoSmFDZ2E2UnRzWERPcnpZVEk0TzJ1RnoyVDhwUVFndC9HZHRVMk01V0ZW?=
 =?utf-8?B?NzkwaUREems4eWNBS09Nekp6Sm1oQmQ4TFhQMjRLRmRJTWkrREwzSElWRFkr?=
 =?utf-8?B?a2QvRTZNaHJqSkwzK1RkOUpDWXhEYUt4YStpcWFCdFdXUDhzcnlGNnFFNDQy?=
 =?utf-8?Q?sN/eBFB0LSNIxlHp/oGuydqbH?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b4962e36-91fc-4862-99a4-08dbfd154566
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Dec 2023 02:26:40.8408
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: D4CavTFsPOvrE/V7yzFuFtU7FXIT1nXq9Ola7RjWjOayezTWweAv5dYOvRuiLjRXFy0zaZZO/3Cb37xUPPtS9Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7017
X-OriginatorOrg: intel.com


On 12/11/2023 11:56 PM, David Hildenbrand wrote:
> Let's mimic what we did with folio_add_file_rmap_*() so we can similarly
> replace page_add_anon_rmap() next.
> 
> Make the compiler always special-case on the granularity by using
> __always_inline.
> 
> Note that the new functions ignore the RMAP_COMPOUND flag, which we will
> remove as soon as page_add_anon_rmap() is gone.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Yin Fengwei <fengwei.yin@intel.com>

With a small question below.

> ---
>   include/linux/rmap.h |   6 +++
>   mm/rmap.c            | 118 ++++++++++++++++++++++++++++++-------------
>   2 files changed, 88 insertions(+), 36 deletions(-)
> 
> diff --git a/include/linux/rmap.h b/include/linux/rmap.h
> index 7198905dc8be..3b5357cb1c09 100644
> --- a/include/linux/rmap.h
> +++ b/include/linux/rmap.h
> @@ -234,6 +234,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio,
>    * rmap interfaces called when adding or removing pte of page
>    */
>   void folio_move_anon_rmap(struct folio *, struct vm_area_struct *);
> +void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages,
> +		struct vm_area_struct *, unsigned long address, rmap_t flags);
> +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \
> +	folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags)
> +void folio_add_anon_rmap_pmd(struct folio *, struct page *,
> +		struct vm_area_struct *, unsigned long address, rmap_t flags);
>   void page_add_anon_rmap(struct page *, struct vm_area_struct *,
>   		unsigned long address, rmap_t flags);
>   void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
> diff --git a/mm/rmap.c b/mm/rmap.c
> index c5761986a411..7787499fa2ad 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -1300,38 +1300,20 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
>   		unsigned long address, rmap_t flags)
>   {
>   	struct folio *folio = page_folio(page);
> -	atomic_t *mapped = &folio->_nr_pages_mapped;
> -	int nr = 0, nr_pmdmapped = 0;
> -	bool compound = flags & RMAP_COMPOUND;
> -	bool first;
>   
> -	/* Is page being mapped by PTE? Is this its first map to be added? */
> -	if (likely(!compound)) {
> -		first = atomic_inc_and_test(&page->_mapcount);
> -		nr = first;
> -		if (first && folio_test_large(folio)) {
> -			nr = atomic_inc_return_relaxed(mapped);
> -			nr = (nr < COMPOUND_MAPPED);
> -		}
> -	} else if (folio_test_pmd_mappable(folio)) {
> -		/* That test is redundant: it's for safety or to optimize out */
> +	if (likely(!(flags & RMAP_COMPOUND)))
> +		folio_add_anon_rmap_pte(folio, page, vma, address, flags);
> +	else
> +		folio_add_anon_rmap_pmd(folio, page, vma, address, flags);
> +}
>   
> -		first = atomic_inc_and_test(&folio->_entire_mapcount);
> -		if (first) {
> -			nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped);
> -			if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) {
> -				nr_pmdmapped = folio_nr_pages(folio);
> -				nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED);
> -				/* Raced ahead of a remove and another add? */
> -				if (unlikely(nr < 0))
> -					nr = 0;
> -			} else {
> -				/* Raced ahead of a remove of COMPOUND_MAPPED */
> -				nr = 0;
> -			}
> -		}
> -	}
> +static __always_inline void __folio_add_anon_rmap(struct folio *folio,
> +		struct page *page, int nr_pages, struct vm_area_struct *vma,
> +		unsigned long address, rmap_t flags, enum rmap_mode mode)
> +{
> +	unsigned int i, nr, nr_pmdmapped = 0;
>   
> +	nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped);
>   	if (nr_pmdmapped)
>   		__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped);
>   	if (nr)
> @@ -1345,18 +1327,34 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
>   		 * folio->index right when not given the address of the head
>   		 * page.
>   		 */
> -		VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio);
> +		VM_WARN_ON_FOLIO(folio_test_large(folio) &&
> +				 mode != RMAP_MODE_PMD, folio);
>   		__folio_set_anon(folio, vma, address,
>   				 !!(flags & RMAP_EXCLUSIVE));
>   	} else if (likely(!folio_test_ksm(folio))) {
>   		__page_check_anon_rmap(folio, page, vma, address);
>   	}
> -	if (flags & RMAP_EXCLUSIVE)
> -		SetPageAnonExclusive(page);
> -	/* While PTE-mapping a THP we have a PMD and a PTE mapping. */
> -	VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 ||
> -			  (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) &&
> -			 PageAnonExclusive(page), folio);
> +
> +	if (flags & RMAP_EXCLUSIVE) {
> +		switch (mode) {
> +		case RMAP_MODE_PTE:
> +			for (i = 0; i < nr_pages; i++)
> +				SetPageAnonExclusive(page + i);
> +			break;
> +		case RMAP_MODE_PMD:
> +			SetPageAnonExclusive(page);
> +			break;
> +		}
> +	}
> +	for (i = 0; i < nr_pages; i++) {
> +		struct page *cur_page = page + i;
> +
> +		/* While PTE-mapping a THP we have a PMD and a PTE mapping. */
> +		VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 ||
> +				  (folio_test_large(folio) &&
> +				   folio_entire_mapcount(folio) > 1)) &&
> +				 PageAnonExclusive(cur_page), folio);
> +	}
This change will iterate all pages for PMD case. The original behavior
didn't check all pages. Is this change by purpose? Thanks.

>   
>   	/*
>   	 * For large folio, only mlock it if it's fully mapped to VMA. It's
> @@ -1368,6 +1366,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
>   		mlock_vma_folio(folio, vma);
>   }
>   
> +/**
> + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio
> + * @folio:	The folio to add the mappings to
> + * @page:	The first page to add
> + * @nr_pages:	The number of pages which will be mapped
> + * @vma:	The vm area in which the mappings are added
> + * @address:	The user virtual address of the first page to map
> + * @flags:	The rmap flags
> + *
> + * The page range of folio is defined by [first_page, first_page + nr_pages)
> + *
> + * The caller needs to hold the page table lock, and the page must be locked in
> + * the anon_vma case: to serialize mapping,index checking after setting,
> + * and to ensure that an anon folio is not being upgraded racily to a KSM folio
> + * (but KSM folios are never downgraded).
> + */
> +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page,
> +		int nr_pages, struct vm_area_struct *vma, unsigned long address,
> +		rmap_t flags)
> +{
> +	__folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags,
> +			      RMAP_MODE_PTE);
> +}
> +
> +/**
> + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio
> + * @folio:	The folio to add the mapping to
> + * @page:	The first page to add
> + * @vma:	The vm area in which the mapping is added
> + * @address:	The user virtual address of the first page to map
> + * @flags:	The rmap flags
> + *
> + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR)
> + *
> + * The caller needs to hold the page table lock, and the page must be locked in
> + * the anon_vma case: to serialize mapping,index checking after setting.
> + */
> +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page,
> +		struct vm_area_struct *vma, unsigned long address, rmap_t flags)
> +{
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> +	__folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags,
> +			      RMAP_MODE_PMD);
> +#else
> +	WARN_ON_ONCE(true);
> +#endif
> +}
> +
>   /**
>    * folio_add_new_anon_rmap - Add mapping to a new anonymous folio.
>    * @folio:	The folio to add the mapping to.