Received-SPF: pass (google.com: domain of linux-kernel+bounces-106005-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <e9e614cd-1df6-4f8b-baef-aaa0a041ec07@linux.alibaba.com>
Date: Mon, 18 Mar 2024 18:31:10 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH v2] mm: support multi-size THP numa balancing
To: David Hildenbrand <david@redhat.com>, "Huang, Ying" <ying.huang@intel.com>
Cc: akpm@linux-foundation.org, mgorman@techsingularity.net,
 wangkefeng.wang@huawei.com, jhubbard@nvidia.com, 21cnbao@gmail.com,
 ryan.roberts@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org
References: <903bf13fc3e68b8dc1f256570d78b55b2dd9c96f.1710493587.git.baolin.wang@linux.alibaba.com>
 <871q88vzc4.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <f00316da-e45d-46c1-99b8-ee160303eaaa@linux.alibaba.com>
 <ca6cca00-8a1b-48c8-b93a-99a772103b8e@redhat.com>
 <3bf2c3e1-44fd-4bc8-a97b-9da7b606aec0@linux.alibaba.com>
 <8e13bce5-e353-4258-9891-97158b8ccd84@redhat.com>
From: Baolin Wang <baolin.wang@linux.alibaba.com>
In-Reply-To: <8e13bce5-e353-4258-9891-97158b8ccd84@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 2024/3/18 18:15, David Hildenbrand wrote:
> On 18.03.24 11:13, Baolin Wang wrote:
>>
>>
>> On 2024/3/18 17:48, David Hildenbrand wrote:
>>> On 18.03.24 10:42, Baolin Wang wrote:
>>>>
>>>>
>>>> On 2024/3/18 14:16, Huang, Ying wrote:
>>>>> Baolin Wang <baolin.wang@linux.alibaba.com> writes:
>>>>>
>>>>>> Now the anonymous page allocation already supports multi-size THP
>>>>>> (mTHP),
>>>>>> but the numa balancing still prohibits mTHP migration even though it
>>>>>> is an
>>>>>> exclusive mapping, which is unreasonable. Thus let's support the
>>>>>> exclusive
>>>>>> mTHP numa balancing firstly.
>>>>>>
>>>>>> Allow scanning mTHP:
>>>>>> Commit 859d4adc3415 ("mm: numa: do not trap faults on shared data
>>>>>> section
>>>>>> pages") skips shared CoW pages' NUMA page migration to avoid shared
>>>>>> data
>>>>>> segment migration. In addition, commit 80d47f5de5e3 ("mm: don't 
>>>>>> try to
>>>>>> NUMA-migrate COW pages that have other uses") change to use
>>>>>> page_count()
>>>>>> to avoid GUP pages migration, that will also skip the mTHP numa
>>>>>> scaning.
>>>>>> Theoretically, we can use folio_maybe_dma_pinned() to detect the GUP
>>>>>> issue, although there is still a GUP race, the issue seems to have 
>>>>>> been
>>>>>> resolved by commit 80d47f5de5e3. Meanwhile, use the
>>>>>> folio_estimated_sharers()
>>>>>> to skip shared CoW pages though this is not a precise sharers 
>>>>>> count. To
>>>>>> check if the folio is shared, ideally we want to make sure every
>>>>>> page is
>>>>>> mapped to the same process, but doing that seems expensive and using
>>>>>> the estimated mapcount seems can work when running autonuma 
>>>>>> benchmark.
>>>>>>
>>>>>> Allow migrating mTHP:
>>>>>> As mentioned in the previous thread[1], large folios are more
>>>>>> susceptible
>>>>>> to false sharing issues, leading to pages ping-pong back and forth
>>>>>> during
>>>>>> numa balancing, which is currently hard to resolve. Therefore, as a
>>>>>> start to
>>>>>> support mTHP numa balancing, only exclusive mappings are allowed to
>>>>>> perform
>>>>>> numa migration to avoid the false sharing issues with large folios.
>>>>>> Similarly,
>>>>>> use the estimated mapcount to skip shared mappings, which seems can
>>>>>> work
>>>>>> in most cases (?), and we've used folio_estimated_sharers() to skip
>>>>>> shared
>>>>>> mappings in migrate_misplaced_folio() for numa balancing, seems no 
>>>>>> real
>>>>>> complaints.
>>>>>
>>>>> IIUC, folio_estimated_sharers() cannot identify multi-thread
>>>>> applications.  If some mTHP is shared by multiple threads in one
>>>>
>>>> Right.
>>>>
>>>
>>> Wasn't this "false sharing" previously raised/described by Mel in this
>>> context?
>>
>> Yes, I got confused with the process's false sharing.
>>
>>>>> process, how to deal with that?
>>>>
>>>> IMHO, seems the should_numa_migrate_memory() already did something to
>>>> help?
>>>>
>>>> ......
>>>>      if (!cpupid_pid_unset(last_cpupid) &&
>>>>                  cpupid_to_nid(last_cpupid) != dst_nid)
>>>>          return false;
>>>>
>>>>      /* Always allow migrate on private faults */
>>>>      if (cpupid_match_pid(p, last_cpupid))
>>>>          return true;
>>>> ......
>>>>
>>>> If the node of the CPU that accessed the mTHP last time is different
>>>> from this time, which means there is some contention for that mTHP 
>>>> among
>>>> threads. So it will not allow migration.
>>>>
>>>> If the contention for the mTHP among threads is light or the accessing
>>>> is relatively stable, then we can allow migration?
>>>>
>>>>> For example, I think that we should avoid to migrate on the first 
>>>>> fault
>>>>> for mTHP in should_numa_migrate_memory().
>>>>>
>>>>> More thoughts?  Can we add a field in struct folio for mTHP to count
>>>>> hint page faults from the same node?
>>>>
>>>> IIUC, we do not need add a new field for folio, seems we can reuse
>>>> ->_flags_2a field. But how to use it? If there are multiple consecutive
>>>> NUMA faults from the same node, then allow migration?
>>>
>>> _flags_2a cannot be used. You could place something after _deferred_list
>>
>> Could you be more explicit? I didn't see _flags_2 currently being used,
>> did I miss something?
> 
> Yes, that we use it implicitly via page->flags on subpages (for example, 
> some flags are still per-subpage and not per-folio).

Yes, thanks for reminding:)