Received-SPF: pass (google.com: domain of linux-kernel+bounces-93-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <6bf0d9c3-23dd-4ab8-b96d-cbab9eeba190@oracle.com>
Date: Thu, 14 Dec 2023 12:51:44 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] mm: remove redundant lru_add_drain() prior to unmapping
 pages
Content-Language: en-US
To: Tim Chen <tim.c.chen@linux.intel.com>, akpm@linux-foundation.org
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org
References: <20231213072805.74201-1-jianfeng.w.wang@oracle.com>
 <3c7d9b8878220571cb7e0760c3a463951252b762.camel@linux.intel.com>
 <d9284139-e32e-493c-86ea-77130b503a77@oracle.com>
 <962b75502b6456a8b698a4ca89d6deedec118ef6.camel@linux.intel.com>
From: Jianfeng Wang <jianfeng.w.wang@oracle.com>
In-Reply-To: <962b75502b6456a8b698a4ca89d6deedec118ef6.camel@linux.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?Z1l1b2M2azRtek9aeFYwSWp1SHhPaEdvb2lwWWJvVDBOZG1YRUpSb0JwRm0x?=
 =?utf-8?B?cnQ4UjBOQkFvRjFOV2ZMK3hNa2ZuYXdWT1lBS1l6ZVJWMTFFNkQ4c3dUMUVO?=
 =?utf-8?B?T2xJeWFiQWd3djUxdGFUakc3R09pWUVoZEdtZWs4RWJEUXdHRWJZT0R4c3Q3?=
 =?utf-8?B?VXgxRlZZbHV5UGF0Z1dmekozeUpndG9wRkxMMDJmT3BKY0hRa0xHRnAxV3Fk?=
 =?utf-8?B?MTFGOEViTWUvNzYvb1h3R1pRamRoaXhuTS8yRUlnc3hLbUdmMFc4NHNQSysy?=
 =?utf-8?B?ZHI4OHllc1ZJbW5NbndtM1VMRFIzT09BNnhqUDdVeGxtM1ZtOENIMWhHeE83?=
 =?utf-8?B?UmVqa2hGOFkzdHc5RzlWZ0l5aWtxOS9PYUllTlZGdGNKeFA5d1R2RiswemZF?=
 =?utf-8?B?citIMGpUeVlNZE5JMVFVcWQ2bjdlclZSV3VGVUxpUW9uQThLUVJZVk1SQ3pC?=
 =?utf-8?B?ais4S3o0RzRmbGsyaWR6UzJ6Z2xtemV4S2dIRVdxa1FtUkE3RDVRdmVEUTE0?=
 =?utf-8?B?cXhzWUhheVRvaGdybUszV3N4VXBOZWFMOTVqOXU4ZkZqaE0vTVBaKzRSbUFq?=
 =?utf-8?B?a0pBWEtpb2NvOFpJOWNVZHVzSWFZa3RvZ1p5KzJWL3lpWWNpNjYwY1ROWkZq?=
 =?utf-8?B?dS8xREtCeGpKdTRDc1UvK09FeTRpaTYzbitlQjdVRGkxMUxQTFZOQTNoVHhX?=
 =?utf-8?B?elNCOE5UQUJES3dsYzRaOE1NV1p0UGIvNE5QQ2wzRUkrcnMyL1BQb2krL3Rw?=
 =?utf-8?B?aHhlT0daYTdUb2orL3NublF1VjhiZ1AzQm1GZ2tPQTIxU2l1cUdMbG0zenpO?=
 =?utf-8?B?cERQcnp6WnhNa0ZQcm9GZS9QS2dueGIrRlQyNUhLWDZOdE9iUWphM3VxYjM5?=
 =?utf-8?B?WHk3dldQWXhHUE9EWkJITkNLMWEyK1Z5V1RpWEo2UFlHNzFmSXR4K1Q0eXMz?=
 =?utf-8?B?dng4eElZREVMTjhYK1NyV0lLTVNuazkvcW9WOFlrMGJDeGc2SWxuanlVZWJ2?=
 =?utf-8?B?OThJd2w5Zk5iWHBnNS9aYjEzNm9FYVV0ZWJHd2MzSnkvb2ZwS3YxSThpRkdl?=
 =?utf-8?B?OThjTmNJbGdhVjRibTdxbUhCdnRVSjNLK1Z0QlVNcG05UlVjd3pEN2xuRXZy?=
 =?utf-8?B?ekJzWlp6NFBNOW9Wd2VTTVREV08xNjIwQ0g0NjB3QzdRaWxGZHBlSGxHeXdh?=
 =?utf-8?B?UW9vbFIwRGM3WnFJUzExUjFtakt2empWV1lnQXBIc0ZZTFUvaU01Nk53U09m?=
 =?utf-8?B?YlFNbTJzWUZCajlHZVRHdGE0NUl3V3ZrU1p3c0UwbmcyOE9IL21ZZlk0aXpO?=
 =?utf-8?B?WVRnelVUOWtaZDVCSG1UdVNKM056NHI1Q0hFcVdkYmlEU2YxWGZGQThqd1ph?=
 =?utf-8?B?QW5zQmEwMkJ4ZDl4a25SY2VGeEYvUDJyellINlpNT2crcXpVaEZKSmFCdUs1?=
 =?utf-8?B?NGZCcDhIV3N5UkZNVG81NHZBMGRlb292RkUyL2pxNHpWWnRQcDZqcHMvTTVI?=
 =?utf-8?B?dU5nV3RXNm13NXNkeUViaE55cFp5UlplaWJZNTU1cFhHa3IweHoyOW9RK1Fz?=
 =?utf-8?B?cFRMcE9vaUZJMmVnYXAwM1R5aWVrY1hTRXhvaDZwZ3ZhSEVYMVE0U3pzTHZH?=
 =?utf-8?B?RG5GOGcyUHdoN2xSMkZkVkJDL3Fza1JwMDFzbEVWYUtKYkNTMEd5VjZlTjd2?=
 =?utf-8?B?U09Xb2E3V2ZhckRvWkgwQ290bm0vTkEwTkxkNThELytLKzR6dVBVTVFyamFM?=
 =?utf-8?B?RHQyUTU0SnlQYmFSKzluNG1yQ05SYzVGYnp0ZzhoaFEzYTZEMEZuRjBoR20x?=
 =?utf-8?B?cXVRekl5RzdHbWtzbGZWTVk0UXBsWjVVZVVWTHpueWZJQ1o3bkl2cEZCZGY0?=
 =?utf-8?B?RFczOW5TQUlneVZWS0J6RGVqQ3JKZTlvRFVyWU1EczZiUGFjL3ZpNUttSVhO?=
 =?utf-8?B?NzVMa3pqaDV5cjBRYmFXZzc2aUdRVlkyNmNUYnpuOHp2dkpDRG03QmtsbTVh?=
 =?utf-8?B?SHR2Sm1jWWpPWVlONGU5cHd4b1p4SWdubXJCeFFDZ0hvU1AvanpFTDJDaFRF?=
 =?utf-8?B?NXVVK0QweEVOS3h0ZlJXbzhIMmtvS2tHT0xSL0Nhb2FGRnppTGhpYVpBa1VW?=
 =?utf-8?B?ZkRCb1hiQnBlamVMN0ZXNTFLeWhvT3FVYThlVlJOQy9weEZSYlFQMHBlSmNC?=
 =?utf-8?B?ZGc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	YgSqq25JLDqIa4Q3oYecdggpSAAbs1+jyFZ22wz0zOref0WegL8RzD6gJ3peq8Z2EMSsT2e/u4MtN/YZNYAyIViSX5sntPvJ6evn8egMbbVDGd95FKgQTy8dM7iE1fhT+4alv8rtQMBJTNlt6NEnILqo7qgH7pbhcluo4P8JuEUPxg4BWtI928HFcmq0njBxfbWUUMAeMysTIwAk32fonjsFq6/866dK16zrOaS0Rv+8/RD1E1PDkwI6z5CU6Fxqtg6qscdOf4RfZKZPCJ3VA5YA4a5lPimCl37siJZ3QZwKh2SbNDEQIzjpieSJ7wPrycfDevsNEk4GdS0wnjfhmbvWBSb6sZ1DaobSREN9q+IfP8miW+4p+n9sarF5tQ7rvrYyZRZgAGBDxkt05tj81s+UCNUry3qwbAxjrKOJ4DVQIw3Pao6VrdABusNrqDGb/ZUN6hNWG/BTBLEycwwCdXTEVOMHtluuGN62dB/RqQpqCAN6iPMvV1S6xjqniRAwXhM5lR0ZEZwzNVaIBC2MeAV2A2+8Vlwp2DcPPoRBXGpbj5tZXalqzRVd5rR+Jm1/a4236poMP9nX0a+0O0raTaZx/GK4ClQ1z53LqqeNl/k=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1381113b-1c7d-40d9-5d73-08dbfce67cea
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB3564.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2023 20:51:47.4367
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uYPbvHZOgguE6/ITXvYypcqDrijxbkaXneZsLsuIR8cdoQmzzMa+0S/qVFXfBBP/uipOqvrJtedYf3ffY0Rn+tJtanYKznrKSbIvITOgog0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4414
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-14_14,2023-12-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 spamscore=0
 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312140147
X-Proofpoint-GUID: ZgHmvnIqujgJRvS2mxqMF75czoqNO5Ym
X-Proofpoint-ORIG-GUID: ZgHmvnIqujgJRvS2mxqMF75czoqNO5Ym

On 12/14/23 9:57 AM, Tim Chen wrote:
> On Wed, 2023-12-13 at 17:03 -0800, Jianfeng Wang wrote:
>> On 12/13/23 2:57 PM, Tim Chen wrote:
>>> On Tue, 2023-12-12 at 23:28 -0800, Jianfeng Wang wrote:
>>>> When unmapping VMA pages, pages will be gathered in batch and released by
>>>> tlb_finish_mmu() if CONFIG_MMU_GATHER_NO_GATHER is not set. The function
>>>> tlb_finish_mmu() is responsible for calling free_pages_and_swap_cache(),
>>>> which calls lru_add_drain() to drain cached pages in folio_batch before
>>>> releasing gathered pages. Thus, it is redundant to call lru_add_drain()
>>>> before gathering pages, if CONFIG_MMU_GATHER_NO_GATHER is not set.
>>>>
>>>> Remove lru_add_drain() prior to gathering and unmapping pages in
>>>> exit_mmap() and unmap_region() if CONFIG_MMU_GATHER_NO_GATHER is not set.
>>>>
>>>> Note that the page unmapping process in oom_killer (e.g., in
>>>> __oom_reap_task_mm()) also uses tlb_finish_mmu() and does not have
>>>> redundant lru_add_drain(). So, this commit makes the code more consistent.
>>>>
>>>> Signed-off-by: Jianfeng Wang <jianfeng.w.wang@oracle.com>
>>>> ---
>>>>  mm/mmap.c | 4 ++++
>>>>  1 file changed, 4 insertions(+)
>>>>
>>>> diff --git a/mm/mmap.c b/mm/mmap.c
>>>> index 1971bfffcc03..0451285dee4f 100644
>>>> --- a/mm/mmap.c
>>>> +++ b/mm/mmap.c
>>>> @@ -2330,7 +2330,9 @@ static void unmap_region(struct mm_struct *mm, struct ma_state *mas,
>>>>  	struct mmu_gather tlb;
>>>>  	unsigned long mt_start = mas->index;
>>>>  
>>>> +#ifdef CONFIG_MMU_GATHER_NO_GATHER
>>>
>>> In your comment you say skip lru_add_drain() when CONFIG_MMU_GATHER_NO_GATHER
>>> is *not* set.  So shouldn't this be
>>>
>>> #ifndef CONFIG_MMU_GATHER_NO_GATHER ?
>>>
>> Hi Tim,
>>
>> The mmu_gather feature is used to gather pages produced by unmap_vmas() and
>> release them in batch in tlb_finish_mmu(). The feature is *on* if
>> CONFIG_MMU_GATHER_NO_GATHER is *not* set. Note that: tlb_finish_mmu() will call
>> free_pages_and_swap_cache()/lru_add_drain() only when the feature is on.
> 
> Thanks for the explanation.
> 
> Looking at the code, lru_add_drain() is executed for #ifndef CONFIG_MMU_GATHER_NO_GATHER
> in tlb_finish_mmu(). So the logic of your patch is fine.
>  
> The #ifndef CONFIG_MMU_GATHER_NO_GATHER means
> mmu_gather feature is on.  The double negative throws me off on in my first read
> of your commit log.
> 
> Suggest that you add a comment in code to make it easier for
> future code maintenence:
> 
> /* defer lru_add_drain() to tlb_finish_mmu() for ifndef CONFIG_MMU_GATHER_NO_GATHER */
> 
> Is your change of skipping the extra lru_add_drain() motivated by some performance reason
> in a workload? Wonder whether it is worth adding an extra ifdef in the code.
> 
> Tim
> 

Okay, great suggestion.

We observe heavy contention on the LRU lock, introduced by lru_add_drain() and
release_pages() for a prod workload, and we're trying to reduce the level of
contention.

lru_add_drain() is a complex function that first takes a local CPU lock and
iterate through *all* folio_batches to see if there are pages to be moved to
and between LRU lists. At that point, any page in these folio_batches will
trigger acquiring the per-LRU spin lock and increase the level of lock
contention. Applying the change can avoid calling lru_add_drain() unnecessarily,
which is a source of lock contention. Together with the comment line suggested
by you, I believe this also increases code readability to clarify the
mmu_gather feature.

- Jianfeng

>>
>> Yes, this commit aims to skip lru_add_drain() when CONFIG_MMU_GATHER_NO_GATHER
>> is *not* set (i.e. when the mmu_gather feature is on) because it is redundant. 
>>
>> If CONFIG_MMU_GATHER_NO_GATHER is set, pages will be released in unmap_vmas().
>> tlb_finish_mmu() will not call lru_add_drain(). So, it is still necessary to
>> keep the lru_add_drain() call to clear cached pages before unmap_vmas(), as
>> folio_batchs hold a reference count for pages in them.
>>
>> The same applies to the other case.
>>
>> Thanks,
>> - Jianfeng
>>
>>>>  	lru_add_drain();
>>>> +#endif
>>>>  	tlb_gather_mmu(&tlb, mm);
>>>>  	update_hiwater_rss(mm);
>>>>  	unmap_vmas(&tlb, mas, vma, start, end, tree_end, mm_wr_locked);
>>>> @@ -3300,7 +3302,9 @@ void exit_mmap(struct mm_struct *mm)
>>>>  		return;
>>>>  	}
>>>>  
>>>> +#ifdef CONFIG_MMU_GATHER_NO_GATHER
>>>
>>> same question as above.
>>>
>>>>  	lru_add_drain();
>>>> +#endif
>>>>  	flush_cache_mm(mm);
>>>>  	tlb_gather_mmu_fullmm(&tlb, mm);
>>>>  	/* update_hiwater_rss(mm) here? but nobody should be looking */
>>>
>