Received-SPF: pass (google.com: domain of linux-kernel+bounces-142142-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: "Huang, Ying" <ying.huang@intel.com>
To: Bharata B Rao <bharata@amd.com>
Cc: <linux-mm@kvack.org>,  <linux-kernel@vger.kernel.org>,
  <akpm@linux-foundation.org>,  <mingo@redhat.com>,
  <peterz@infradead.org>,  <mgorman@techsingularity.net>,
  <raghavendra.kt@amd.com>,  <dave.hansen@linux.intel.com>,
  <hannes@cmpxchg.org>
Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large
 address space
In-Reply-To: <8692d514-d1c8-4fbf-84d7-1ad609480070@amd.com> (Bharata B. Rao's
	message of "Fri, 12 Apr 2024 09:30:45 +0530")
References: <20240327160237.2355-1-bharata@amd.com>
	<87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<dd2bc563-7654-4d83-896e-49a7291dd1aa@amd.com>
	<87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com>
	<875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com>
	<87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<9ec3b04b-bde8-42ce-be1b-34f7d8e6762d@amd.com>
	<87il0yet4z.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<8692d514-d1c8-4fbf-84d7-1ad609480070@amd.com>
Date: Fri, 12 Apr 2024 15:28:16 +0800
Message-ID: <87cyqv59bj.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13)
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii

Bharata B Rao <bharata@amd.com> writes:

> On 03-Apr-24 2:10 PM, Huang, Ying wrote:
>> Bharata B Rao <bharata@amd.com> writes:
>> 
>>> On 02-Apr-24 7:33 AM, Huang, Ying wrote:
>>>> Bharata B Rao <bharata@amd.com> writes:
>>>>
>>>>> On 29-Mar-24 6:44 AM, Huang, Ying wrote:
>>>>>> Bharata B Rao <bharata@amd.com> writes:
>>>>> <snip>
>>>>>>> I don't think the pages are cold but rather the existing mechanism fails
>>>>>>> to categorize them as hot. This is because the pages were scanned way
>>>>>>> before the accesses start happening. When repeated accesses are made to
>>>>>>> a chunk of memory that has been scanned a while back, none of those
>>>>>>> accesses get classified as hot because the scan time is way behind
>>>>>>> the current access time. That's the reason we are seeing the value
>>>>>>> of latency ranging from 20s to 630s as shown above.
>>>>>>
>>>>>> If repeated accesses continue, the page will be identified as hot when
>>>>>> it is scanned next time even if we don't expand the threshold range.  If
>>>>>> the repeated accesses only last very short time, it makes little sense
>>>>>> to identify the pages as hot.  Right?
>>>>>
>>>>> The total allocated memory here is 192G and the chunk size is 1G. Each
>>>>> time one such 1G chunk is taken up randomly for generating memory accesses.
>>>>> Within that 1G, 262144 random accesses are performed and 262144 such
>>>>> accesses are repeated for 512 times. I thought that should be enough
>>>>> to classify that chunk of memory as hot.
>>>>
>>>> IIUC, some pages are accessed in very short time (maybe within 1ms).
>>>> This isn't repeated access in a long period.  I think that pages
>>>> accessed repeatedly in a long period are good candidates for promoting.
>>>> But pages accessed frequently in only very short time aren't.
>>>
>>> Here are the numbers for the 192nd chunk:
>>>
>>> Each iteration of 262144 random accesses takes around ~10ms
>>> 512 such iterations are taking ~5s
>>> numa_scan_seq is 16 when this chunk is accessed.
>>> And no page promotions were done from this chunk. All the
>>> time should_numa_migrate_memory() found the NUMA hint fault
>>> latency to be higher than threshold.
>>>
>>> Are these time periods considered too short for the pages
>>> to be detected as hot and promoted?
>> 
>> Yes.  I think so.  This is burst accessing, not repeated accessing.
>> IIUC, NUMA balancing based promotion only works for repeated accessing
>> for long time, for example, >100s.
>
> Hmm... When a page is accessed 512 times over a period of 5s and it is
> still not detected as hot. This is understandable if fresh scanning couldn't
> be done as the accesses were bursty and hence they couldn't be captured via
> NUMA hint faults. But here the access captured via hint fault is being rejected
> as not hot because the scanning was done a while back. But I do see the challenge
> here since we depend on scanning time to obtain the frequency-of-access metric.

Consider some pages that will be accessed once every 1 hour, should we
consider it hot or not?  Will your proposed method deal with that
correctly?

> BTW, for the above same scenario with numa_balancing_mode=1, the remote
> accesses get detected and migration to source node is tried. It is a different
> matter that eventually pages can't be migrated in this specific scenario as
> the src node is already full.

--
Best Regards,
Huang, Ying