Received-SPF: pass (google.com: domain of linux-kernel+bounces-82818-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <009e5633-decb-4c21-b5fc-58984fbade96@intel.com>
Date: Tue, 27 Feb 2024 15:42:32 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/1] mm/madvise: enhance lazyfreeing with mTHP in
 madvise_free
Content-Language: en-US
To: Barry Song <21cnbao@gmail.com>
CC: Ryan Roberts <ryan.roberts@arm.com>, Lance Yang <ioworker0@gmail.com>,
	David Hildenbrand <david@redhat.com>, <akpm@linux-foundation.org>,
	<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>, <mhocko@suse.com>,
	<minchan@kernel.org>, <peterx@redhat.com>, <shy828301@gmail.com>,
	<songmuchun@bytedance.com>, <wangkefeng.wang@huawei.com>,
	<zokeefe@google.com>
References: <CAGsJ_4zera4+bWuXHKjdU3QdiR3sqcUQB3gF08DPD49OCT4S9w@mail.gmail.com>
 <20240226083714.26187-1-ioworker0@gmail.com>
 <9bcf5141-7376-441e-bbe3-779956ef28b9@redhat.com>
 <CAK1f24mdwjz2J5VmeYSDwb4jMbrXUVCSF_0pOcWSVt1cfa0FhQ@mail.gmail.com>
 <318be511-06de-423e-8216-af869f27f849@arm.com>
 <CAGsJ_4z+0yXUDYwxKNAqYwxAAYpfKpKd2u_dVTDP3b-KPOQT1g@mail.gmail.com>
 <19758162-be5f-4dc4-b316-77b0115d12ce@intel.com>
 <CAGsJ_4wx72KOazANBvnGcjdZse8W9+PW5_fspP9=QuX3X_7msg@mail.gmail.com>
 <3c56d7b8-b76d-4210-b431-ee6431775ba7@intel.com>
 <CAGsJ_4xu1kz5VD-CcNFvP0A1nPKDojV8Gy1HPvNKuQ_RAw=26g@mail.gmail.com>
 <6ea0020a-8f4b-44d1-a3b2-7c2905d32772@intel.com>
 <CAGsJ_4x6Otb9LUvnxAaPLnQ2MPPng0xpG-vJmFL7pNm10FDhZA@mail.gmail.com>
 <CAGsJ_4xndYM8=7v+EV_aWX+_qgA1UPmm38n+ujbQXJLzCPKfog@mail.gmail.com>
From: Yin Fengwei <fengwei.yin@intel.com>
In-Reply-To: <CAGsJ_4xndYM8=7v+EV_aWX+_qgA1UPmm38n+ujbQXJLzCPKfog@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eGpvcU9VRUxqS0tlN1hkQVBCTmRxdGdFcDUvaHgzWmJKMzhJYndsVDFNdkxS?=
 =?utf-8?B?c3RrekxCSGp1TTRWYTdnanArYktMcGFZaGR5cUJLMUFuUFFYN21Cbm9UbkRI?=
 =?utf-8?B?WUVHZmdZQVRzVG9iVTgrQUpuK3I4NDE0Q3RmYms3OWJJYVhPZDNRdnNDS0Nq?=
 =?utf-8?B?U2hTazlHdWg4aFFZMFExdkpDSEdDRmk3a0VSZlcydzU4M0xNUDVnbFNYNXJI?=
 =?utf-8?B?MGk0TDB6REhDalVNZ0J1RldXanV1VVZOUTRkcnAzblFlT0Z1REtCZzBQYk5N?=
 =?utf-8?B?eDlMK28wZEFLZGFRa3FBSDNKL1hKZ29qaDVIeTJzVDZaRGQ5bWRMcERDM05J?=
 =?utf-8?B?T3p3MWNhTGtBUzRmeTcyanU4cTlIbmI5MUFvYmF0M0VpYzRHZ3U3N0lHL2Vn?=
 =?utf-8?B?N1kwSzhsUzJYRG4zRk9MU3dPQ1VzOGN1czlTelFDYWs2TWVINXI2YmJvdEE0?=
 =?utf-8?B?M2xrVHVoVlAyR0tOck1HekRJUWdSM2ZIVmhlU01PM3RrY2l5eCtteGpkdU1y?=
 =?utf-8?B?YlAwQkFodnpRVkZMOVAvTi8wWDF5NlljYkh5RlRnanBBeXpDcUtseFFXUncz?=
 =?utf-8?B?R3dNMHluRzRVaUZuc21XcWFsZEJtYldBdGRURGNLK1N0ZERJK2pTMi9IUlBI?=
 =?utf-8?B?V1lXTlh6ZWhxN1RsM3BucW5oT2c4TXJkczJWVmxyeHRablJ6TDF1TnZEY05l?=
 =?utf-8?B?SndPaHhWQXZXRndtUk1QTFEzdFcxa1JZVThEUklFSnlTUVZWT0hIWDRZNzVG?=
 =?utf-8?B?cjQzOFpSbVhqbHFUaHptZE8zdzVkVCt5VjM2Y2loblp5Sks1TlN6cDVBVVpU?=
 =?utf-8?B?YWpNbDdTSE9KcVBvc2dYWGRsRjBNY1dWUzljTkhuTUR6RjROTGExQ01NS3Q4?=
 =?utf-8?B?czVxUnlaYkN1ZXRWTGxHK2ZmVEJmbnd0MWlqWFBSRkQvZ1VPZnl5Ni8xeTZQ?=
 =?utf-8?B?QU9SK21PMGtNbzFQaVJCNTlvUFRINGZyMnpOVGVzeWUxSXlzdmJRbHp2ZmVp?=
 =?utf-8?B?ZTcyQ0M4bFREWE5XdmpKbFdGa3RNVlFucFVyYmdyVWIyMU82dmlYSGZEb201?=
 =?utf-8?B?SGZHYVFDN1k2WjltTjFMRjlsZEthRmtqZnVlZWd6ZGVUWnVWcXRmNFRac1pN?=
 =?utf-8?B?WGN1R3pHMEtMRHZBY3hGbG5QVzcxakZhd1IzckVHSVJ1OGp6M1MrL1ZhNXB5?=
 =?utf-8?B?WFZBekNaSWw4citRVFVtQU1QNTEzZ3ZwN1k2M2p6UU1ibFdQTGZOSVI2YUNx?=
 =?utf-8?B?NDgyK0NkVHFNSzl3dEVhWHNyZDFjNlhtc09OdUJVS0tPbVFZSUxmZnRRd3Ey?=
 =?utf-8?B?ZkQ1aUlsMVAvMFBBRlhZYW5rSlE0V0RkdGFiTWhZMU1xSkZDT0xNZ1hnUXI4?=
 =?utf-8?B?K28vVUxyRHBpT2NlU3ZwY2xGT2J6NEZwK2xOby9NamNGeGM0akpsSWVFb05k?=
 =?utf-8?B?NXd6enpuNlpPQjhQWVpSZjB2VTVuWTV6OHM3c2JkRDJZckdCU3Jibm96bEN0?=
 =?utf-8?B?a0dJMTdMYjhMMHFDcCtuUDRIeE1vdTZqMDB2WDlXRE9MdHlENUFPaXBlSXpK?=
 =?utf-8?B?Q2huLzRIODhtcjkyWTFIUGdIZy9sbWgxRUNReWNxUk9TVWFReU9odnI0T1ZG?=
 =?utf-8?B?Y3UxY09hSVdGc1JuTldvSmRkcEhzRXdoVXB2c3ltcU9ZbFhVMTg1MjVac25S?=
 =?utf-8?B?RndoeE1TQnZYcG40emNnelU0cWc2MElmNXBYTTVPaXF1QUlvamJTQWk5WEpa?=
 =?utf-8?B?dzVmdEVLV2xaV21EbUR2NGtRY3BndFJsU0VsT1NmRTRFV1FHcm5EVmlXZnI5?=
 =?utf-8?B?aGhHQkk5SkUrYU8xbzVkcXhyZWlZMHJVa1QvU2hCOHFtalg4eGlKeldiVzQ0?=
 =?utf-8?B?T1ZuY25FVnNDbTRaelRWRCtRWFFyd09oTS9reWFJaTZmazR6STFDTmlxZHlP?=
 =?utf-8?B?L3NaL1lXa2JDQ3RVbjdOU1JaUEMzbFhuc2Jvakx2aFpTbVdoRlBZb1BBV3R2?=
 =?utf-8?B?bHk3aTRBWWRuMFljOUMyQnkwRFJHRUtTZEdaZll1dUhUOXVMQ1J3VGEwOXh5?=
 =?utf-8?B?UDZhd0FQcGlQVlQxWjgrM2xyRmJRMERiM0YrMXdyeVgvMmlnb0t4d1VJUyt4?=
 =?utf-8?B?NERYN0tTMG1GL2kwamtKR2t6OG5GY245RHJRWGYyaGtZRVNlN29RWUlYOXp2?=
 =?utf-8?B?S3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b7857986-7377-49dd-6f96-08dc3767ace1
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2024 07:42:40.2078
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Hil3xbWHj9yEQwoGjs2es4Zh+sWGQwbDph4oMUOB6B/J1PVqwm9/OQc+6yj5qbfFcVXgwbp8d2uafWQurbc71g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR11MB6082
X-OriginatorOrg: intel.com


On 2/27/24 15:21, Barry Song wrote:
> On Tue, Feb 27, 2024 at 8:11 PM Barry Song <21cnbao@gmail.com> wrote:
>>
>> On Tue, Feb 27, 2024 at 8:02 PM Yin Fengwei <fengwei.yin@intel.com> wrote:
>>>
>>>
>>>
>>> On 2/27/24 14:40, Barry Song wrote:
>>>> On Tue, Feb 27, 2024 at 7:14 PM Yin Fengwei <fengwei.yin@intel.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2/27/24 10:17, Barry Song wrote:
>>>>>>> Like if we hit folio which is partially mapped to the range, don't split it but
>>>>>>> just unmap the mapping part from the range. Let page reclaim decide whether
>>>>>>> split the large folio or not (If it's not mapped to any other range,it will be
>>>>>>> freed as whole large folio. If part of it still mapped to other range,page reclaim
>>>>>>> can decide whether to split it or ignore it for current reclaim cycle).
>>>>>> Yes, we can. but we still have to play the ptes check game to avoid adding
>>>>>> folios multiple times to reclaim the list.
>>>>>>
>>>>>> I don't see too much difference between splitting in madvise and splitting
>>>>>> in vmscan.  as our real purpose is avoiding splitting entirely mapped
>>>>>> large folios. for partial mapped large folios, if we split in madvise, then
>>>>>> we don't need to play the game of skipping folios while iterating PTEs.
>>>>>> if we don't split in madvise, we have to make sure the large folio is only
>>>>>> added in reclaimed list one time by checking if PTEs belong to the
>>>>>> previous added folio.
>>>>>
>>>>> If the partial mapped large folio is unmapped from the range, the related PTE
>>>>> become none. How could the folio be added to reclaimed list multiple times?
>>>>
>>>> in case we have 16 PTEs in a large folio.
>>>> PTE0 present
>>>> PTE1 present
>>>> PTE2 present
>>>> PTE3  none
>>>> PTE4 present
>>>> PTE5 none
>>>> PTE6 present
>>>> ....
>>>> the current code is scanning PTE one by one.
>>>> while scanning PTE0, we have added the folio. then PTE1, PTE2, PTE4, PTE6...
>>> No. Before detect the folio is fully mapped to the range, we can't add folio
>>> to reclaim list because the partial mapped folio shouldn't be added. We can
>>> only scan PTE15 and know it's fully mapped.
>>
>> you never know PTE15 is the last one mapping to the large folio, PTE15 can
>> be mapping to a completely different folio with PTE0.
>>
>>>
>>> So, when scanning PTE0, we will not add folio. Then when hit PTE3, we know
>>> this is a partial mapped large folio. We will unmap it. Then all 16 PTEs
>>> become none.
>>
>> I don't understand why all 16PTEs become none as we set PTEs to none.
>> we set PTEs to swap entries till try_to_unmap_one called by vmscan.
>>
>>>
>>> If the large folio is fully mapped, the folio will be added to reclaim list
>>> after scan PTE15 and know it's fully mapped.
>>
>> our approach is calling pte_batch_pte while meeting the first pte, if
>> pte_batch_pte = 16,
>> then we add this folio to reclaim_list and skip the left 15 PTEs.
> 
> Let's compare two different implementation, for partial mapped large folio
> with 8 PTEs as below,
> 
> PTE0 present for large folio1
> PTE1 present for large folio1
> PTE2 present for another folio2
> PTE3 present for another folio3
> PTE4 present for large folio1
> PTE5 present for large folio1
> PTE6 present for another folio4
> PTE7 present for another folio5
> 
> If we don't split in madvise(depend on vmscan to split after adding
> folio1), we will have
Let me clarify something here:

I prefer that we don't split large folio here. Instead, we unmap the
large folio from this VMA range (I think you missed the unmap operation
I mentioned).

The intention is trying best to avoid splitting the large folio. If
the folio is only partially mapped to this VMA range, it's likely it
will be reclaimed as whole large folio. Which brings benefit for lru
and zone lock contention comparing to splitting large folio.

The thing I am not sure is unmapping from specific VMA range is not
available and whether it's worthy to add it.

> to make sure folio1, folio2, folio3, folio4, folio5 are added to
> reclaim_list by doing a complex
> game while scanning these 8 PTEs.
> 
> if we split in madvise, they become:
> 
> PTE0 present for large folioA  - splitted from folio 1
> PTE1 present for large folioB - splitted from folio 1
> PTE2 present for another folio2
> PTE3 present for another folio3
> PTE4 present for large folioC - splitted from folio 1
> PTE5 present for large folioD - splitted from folio 1
> PTE6 present for another folio4
> PTE7 present for another folio5
> 
> we simply add the above 8 folios into reclaim_list one by one.
> 
> I would vote for splitting for partial mapped large folio in madvise.
> 
> Thanks
> Barry