Received-SPF: pass (google.com: domain of linux-kernel+bounces-82868-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <05f2d04c-333d-4298-8c7a-d5adeac5df82@intel.com>
Date: Tue, 27 Feb 2024 16:33:33 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/1] mm/madvise: enhance lazyfreeing with mTHP in
 madvise_free
Content-Language: en-US
To: Barry Song <21cnbao@gmail.com>
CC: Ryan Roberts <ryan.roberts@arm.com>, Lance Yang <ioworker0@gmail.com>,
	David Hildenbrand <david@redhat.com>, <akpm@linux-foundation.org>,
	<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>, <mhocko@suse.com>,
	<minchan@kernel.org>, <peterx@redhat.com>, <shy828301@gmail.com>,
	<songmuchun@bytedance.com>, <wangkefeng.wang@huawei.com>,
	<zokeefe@google.com>
References: <CAGsJ_4zera4+bWuXHKjdU3QdiR3sqcUQB3gF08DPD49OCT4S9w@mail.gmail.com>
 <20240226083714.26187-1-ioworker0@gmail.com>
 <9bcf5141-7376-441e-bbe3-779956ef28b9@redhat.com>
 <CAK1f24mdwjz2J5VmeYSDwb4jMbrXUVCSF_0pOcWSVt1cfa0FhQ@mail.gmail.com>
 <318be511-06de-423e-8216-af869f27f849@arm.com>
 <CAGsJ_4z+0yXUDYwxKNAqYwxAAYpfKpKd2u_dVTDP3b-KPOQT1g@mail.gmail.com>
 <19758162-be5f-4dc4-b316-77b0115d12ce@intel.com>
 <CAGsJ_4wx72KOazANBvnGcjdZse8W9+PW5_fspP9=QuX3X_7msg@mail.gmail.com>
 <3c56d7b8-b76d-4210-b431-ee6431775ba7@intel.com>
 <CAGsJ_4xu1kz5VD-CcNFvP0A1nPKDojV8Gy1HPvNKuQ_RAw=26g@mail.gmail.com>
 <6ea0020a-8f4b-44d1-a3b2-7c2905d32772@intel.com>
 <CAGsJ_4x6Otb9LUvnxAaPLnQ2MPPng0xpG-vJmFL7pNm10FDhZA@mail.gmail.com>
 <CAGsJ_4xndYM8=7v+EV_aWX+_qgA1UPmm38n+ujbQXJLzCPKfog@mail.gmail.com>
 <009e5633-decb-4c21-b5fc-58984fbade96@intel.com>
 <CAGsJ_4yftgfD7MsNd5iej8_wTrYeTsAFfW7xmqrj-Q_w8-oqMg@mail.gmail.com>
From: Yin Fengwei <fengwei.yin@intel.com>
In-Reply-To: <CAGsJ_4yftgfD7MsNd5iej8_wTrYeTsAFfW7xmqrj-Q_w8-oqMg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?d3JjTGpDMU1WanIrUG42Y1dkQ05DdTFzNm1XSk55WTFibWlmc3VQYWdMTEQx?=
 =?utf-8?B?d3lOZDVNVUhOcGtEOW1vdlk0NmdQYVo5QVFrdkZoNUJENHYwQ0xWb3Y5d0dE?=
 =?utf-8?B?LzF2TjJuS3F4TytOTlpZQmtDSUlzYUIydTBCWVJyUEZFVXNTb0FNd21aWWlt?=
 =?utf-8?B?RjVoNjY1anJRb3R3M2RjRHVwcXBaSVc3UzZ4cXR5bGZYdlVrdHFmZnBDQkhF?=
 =?utf-8?B?OXdYNndTZ2cxMEJtOGt4M2dZSWpYY0dITWZZZCthWWVjNm5ha0thUGErc2Uv?=
 =?utf-8?B?UnZZVTRKZkRPbmxtK0pvdHhCOUt3NnZLL0lDa1F0dGwrWHZEalRoM3dOUGw3?=
 =?utf-8?B?V1NEV0JFUzd5NUVFK0lmeCtJTkJtK2FsV0JmL1I3T3V4ZTNnR3p6TU5KbjBV?=
 =?utf-8?B?T2hLbkJyc2hjNzUrcHhOWENRUC9oSVZacWhxcDdzZ1IwZzdMa1MvUGJhT3Y2?=
 =?utf-8?B?U2gxd1JteFZkQVNlUDEyZDV1M0NjM1dEV0pGWDdXdVFYTlhnZjNDeFV1bWc1?=
 =?utf-8?B?YXZEWDV2ejJPMkVSTFY1R0JMcjVkTXlyeHNZZ0NqZytmT0ZGZzRvSW11M2I4?=
 =?utf-8?B?cmR0MkZldTR4ajhJeWI0NjZoMFN4bnUraGs4Z01WMnhDQldiY1ZoS3QwTzNX?=
 =?utf-8?B?MGorc3hZODU5aE0vdXBmaHQyelRJSmNIYmd2Nk5GY0o3c2VSR25OdUgrYWJP?=
 =?utf-8?B?MFVxdExzdlpQU0RmOWU3OEdMWWd6dTdHNUtkQTNaSlZtTXkxc0Vwa2N4bWYz?=
 =?utf-8?B?MDZqN1g2QWVDSmlvMXFIR2ZNWFBOdDdqYjlqVXoxM1BxUnRwZys5TGdseUt0?=
 =?utf-8?B?ak5xSlVyWmtkUlMvWWoyMEhYNk10eEc1VUVNeUZFTjNialN4d1dPdFF6Sjgz?=
 =?utf-8?B?ZURTRUhYZTFHaTVWT0hScGhDeUhRTHlsVThWcjZ5ZFozcHNMS2tFZStpNnJV?=
 =?utf-8?B?azVQWUxsWWd0SUpqb1prY3dhNVhJbGlvdENjTzZMYnd0dXd4MHRHVGZoZDRO?=
 =?utf-8?B?Vk5jcWt4V0JtV3B3cWpGQWlld21KaUtmb1NZT3czZUpSTmJ1VXRRdTFTZDZ0?=
 =?utf-8?B?TytkckRxYzljbHAyWGczZ3ZhVWdCbWE0T3BxRnp0Y0FURmpNcXpoUEg2OEN5?=
 =?utf-8?B?SHNrTFFGUmF1NW9KS1MraW5FOUdHMUM5dlc5R2lzSnRnclRuMFl4YWVSalNT?=
 =?utf-8?B?bEtCNkhQNHdqOCtlR2d0eWsxWld0TVZxbk5mSVN3UUxJSUp1TXZNWE90dDVz?=
 =?utf-8?B?VXpxdDFWaXJPalBTUTF5YjFxSGNCWExrcHFnUlloVldyZFh1QU44eGpSK1lm?=
 =?utf-8?B?ZzJwNWhXQWFWaENvNmEvdGg4ZFljTUJZdC9qNFlMVk44OTdXTnZjRlY3REZ4?=
 =?utf-8?B?ajJZVWxpRU5ONVN5V1BLT2Y0ZE5BcUZKSGxvNTM4U0RHRVRMQWJJeTZPbVNG?=
 =?utf-8?B?UEcwWDVsTmsvUEc4bDJzcE1Oam1xMC9MR2xhK280VFZaSU01VzNyMVFzc2kr?=
 =?utf-8?B?ZldOVXNVVStLSVpsWDg3aVV5NTRUSjJuQnBhUkhTMWRMcWZ3UFdwTW5pOXVk?=
 =?utf-8?B?NVdlOTRkNzllQy9tNjR3bTZxMnIwbWJ0T3VhY0tDSFNhNUhFVE5mS2FtVnNr?=
 =?utf-8?B?UEZ2b1krTUdXMXU1NkJWdzZwQWoyU2NmdEhjSEViZ3hqTEhXNEhHaDhKOGl1?=
 =?utf-8?B?UkMwbEVyZGxadUlmaFJPbldIVmVpTkcrNVVrWTNPSCtlczUzM3lLT2Npdjhu?=
 =?utf-8?B?RlNxZThUM2NkZWliRGVDY0xFVjR6V3RKZkw2NHk0NHhTdzJtNXNVYjFEVDlS?=
 =?utf-8?B?SHl1OWJwKzUxaUVUbTByWFJDcUkzYk90Z2ZscEFFY2x2MVJIOHhqUW9XWVBT?=
 =?utf-8?B?andLd1JUcHkwZ0N0emlodkFtdVlTbFhqRDVtbzFpNWRUZUZLS0NsMENOU0VN?=
 =?utf-8?B?bG40OW9GYUtHakRGMlhhYWdKaDRPZkdMSUVKdUdWZEY4UHZnSlhMcmJaSHVX?=
 =?utf-8?B?cWw5bVp1Q1BPaVJlcFVLWjRlRkhsS1prOHV0SkNBNnRPRDBlSTVrQzk5cDVn?=
 =?utf-8?B?TVJia3NuMDNrTW50eVc1VSszZnlsS0U4QzhObmcvWWJpa3ljVnYzK0JrSUJv?=
 =?utf-8?B?Y1VNU2M0TWVZaVowVDBzQzFnTzg1MGIzeUh1UDN2ZWoyM1lxelJUZWlhWWFX?=
 =?utf-8?B?T1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2832f4c7-41aa-4cc4-7396-08dc376ecccc
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2024 08:33:40.4087
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dbe0mINgjwx8C98rxoaHgreY6p9CW7V8xcMXVLrf8z5GRb5PRNF1bIxldCPAR6ym7Z++FDdZmq2T3lQEjMKwlA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6283
X-OriginatorOrg: intel.com


On 2/27/24 15:54, Barry Song wrote:
> On Tue, Feb 27, 2024 at 8:42 PM Yin Fengwei <fengwei.yin@intel.com> wrote:
>>
>>
>>
>> On 2/27/24 15:21, Barry Song wrote:
>>> On Tue, Feb 27, 2024 at 8:11 PM Barry Song <21cnbao@gmail.com> wrote:
>>>>
>>>> On Tue, Feb 27, 2024 at 8:02 PM Yin Fengwei <fengwei.yin@intel.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2/27/24 14:40, Barry Song wrote:
>>>>>> On Tue, Feb 27, 2024 at 7:14 PM Yin Fengwei <fengwei.yin@intel.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2/27/24 10:17, Barry Song wrote:
>>>>>>>>> Like if we hit folio which is partially mapped to the range, don't split it but
>>>>>>>>> just unmap the mapping part from the range. Let page reclaim decide whether
>>>>>>>>> split the large folio or not (If it's not mapped to any other range,it will be
>>>>>>>>> freed as whole large folio. If part of it still mapped to other range,page reclaim
>>>>>>>>> can decide whether to split it or ignore it for current reclaim cycle).
>>>>>>>> Yes, we can. but we still have to play the ptes check game to avoid adding
>>>>>>>> folios multiple times to reclaim the list.
>>>>>>>>
>>>>>>>> I don't see too much difference between splitting in madvise and splitting
>>>>>>>> in vmscan.  as our real purpose is avoiding splitting entirely mapped
>>>>>>>> large folios. for partial mapped large folios, if we split in madvise, then
>>>>>>>> we don't need to play the game of skipping folios while iterating PTEs.
>>>>>>>> if we don't split in madvise, we have to make sure the large folio is only
>>>>>>>> added in reclaimed list one time by checking if PTEs belong to the
>>>>>>>> previous added folio.
>>>>>>>
>>>>>>> If the partial mapped large folio is unmapped from the range, the related PTE
>>>>>>> become none. How could the folio be added to reclaimed list multiple times?
>>>>>>
>>>>>> in case we have 16 PTEs in a large folio.
>>>>>> PTE0 present
>>>>>> PTE1 present
>>>>>> PTE2 present
>>>>>> PTE3  none
>>>>>> PTE4 present
>>>>>> PTE5 none
>>>>>> PTE6 present
>>>>>> ....
>>>>>> the current code is scanning PTE one by one.
>>>>>> while scanning PTE0, we have added the folio. then PTE1, PTE2, PTE4, PTE6...
>>>>> No. Before detect the folio is fully mapped to the range, we can't add folio
>>>>> to reclaim list because the partial mapped folio shouldn't be added. We can
>>>>> only scan PTE15 and know it's fully mapped.
>>>>
>>>> you never know PTE15 is the last one mapping to the large folio, PTE15 can
>>>> be mapping to a completely different folio with PTE0.
>>>>
>>>>>
>>>>> So, when scanning PTE0, we will not add folio. Then when hit PTE3, we know
>>>>> this is a partial mapped large folio. We will unmap it. Then all 16 PTEs
>>>>> become none.
>>>>
>>>> I don't understand why all 16PTEs become none as we set PTEs to none.
>>>> we set PTEs to swap entries till try_to_unmap_one called by vmscan.
>>>>
>>>>>
>>>>> If the large folio is fully mapped, the folio will be added to reclaim list
>>>>> after scan PTE15 and know it's fully mapped.
>>>>
>>>> our approach is calling pte_batch_pte while meeting the first pte, if
>>>> pte_batch_pte = 16,
>>>> then we add this folio to reclaim_list and skip the left 15 PTEs.
>>>
>>> Let's compare two different implementation, for partial mapped large folio
>>> with 8 PTEs as below,
>>>
>>> PTE0 present for large folio1
>>> PTE1 present for large folio1
>>> PTE2 present for another folio2
>>> PTE3 present for another folio3
>>> PTE4 present for large folio1
>>> PTE5 present for large folio1
>>> PTE6 present for another folio4
>>> PTE7 present for another folio5
>>>
>>> If we don't split in madvise(depend on vmscan to split after adding
>>> folio1), we will have
>> Let me clarify something here:
>>
>> I prefer that we don't split large folio here. Instead, we unmap the
>> large folio from this VMA range (I think you missed the unmap operation
>> I mentioned).
> 
> I don't understand why we unmap as this is a MADV_PAGEOUT not
> an unmap. unmapping totally changes the semantics. Would you like
> to show pseudo code?
Oh. Yes. MADV_PAGEOUT is not suitable.

What about MADV_FREE?

> 
> for MADV_PAGEOUT on swap-out, the last step is writing swap entries
> to replace PTEs which are present. I don't understand how an unmap
> can be involved in this process.
> 
>>
>> The intention is trying best to avoid splitting the large folio. If
>> the folio is only partially mapped to this VMA range, it's likely it
>> will be reclaimed as whole large folio. Which brings benefit for lru
>> and zone lock contention comparing to splitting large folio.
> 
> which also brings negative side effects such as redundant I/O.
> For example, if you have only one subpage left in a large folio,
> pageout will still write nr_pages subpages into swap, then immediately
> free them in swap.
> 
>>
>> The thing I am not sure is unmapping from specific VMA range is not
>> available and whether it's worthy to add it.
> 
> I think we might have the possibility to have some complex code to
> add folio1, folio2, folio3, folio4 and folio5 in the above example into
> reclaim_list while avoiding splitting folio1. but i really don't understand
> how unmap will work.
> 
>>
>>> to make sure folio1, folio2, folio3, folio4, folio5 are added to
>>> reclaim_list by doing a complex
>>> game while scanning these 8 PTEs.
>>>
>>> if we split in madvise, they become:
>>>
>>> PTE0 present for large folioA  - splitted from folio 1
>>> PTE1 present for large folioB - splitted from folio 1
>>> PTE2 present for another folio2
>>> PTE3 present for another folio3
>>> PTE4 present for large folioC - splitted from folio 1
>>> PTE5 present for large folioD - splitted from folio 1
>>> PTE6 present for another folio4
>>> PTE7 present for another folio5
>>>
>>> we simply add the above 8 folios into reclaim_list one by one.
>>>
>>> I would vote for splitting for partial mapped large folio in madvise.
>>>
> 
> Thanks
> Barry