Received-SPF: pass (google.com: domain of linux-kernel+bounces-141883-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <8692d514-d1c8-4fbf-84d7-1ad609480070@amd.com>
Date: Fri, 12 Apr 2024 09:30:45 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address
 space
Content-Language: en-US
To: "Huang, Ying" <ying.huang@intel.com>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
 akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org,
 mgorman@techsingularity.net, raghavendra.kt@amd.com,
 dave.hansen@linux.intel.com, hannes@cmpxchg.org
References: <20240327160237.2355-1-bharata@amd.com>
 <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <dd2bc563-7654-4d83-896e-49a7291dd1aa@amd.com>
 <87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com>
 <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com>
 <87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <9ec3b04b-bde8-42ce-be1b-34f7d8e6762d@amd.com>
 <87il0yet4z.fsf@yhuang6-desk2.ccr.corp.intel.com>
From: Bharata B Rao <bharata@amd.com>
In-Reply-To: <87il0yet4z.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?K0srN1NmV215Sjd2eVlWcDJ2R0JXbFpKVmVZM3ZFZnRKZjFtbWY4OG55eUJk?=
 =?utf-8?B?WS9KcG85bUV3UDRVU3BEOXpuamRLdFJiOVAxVi93OVRHY29nOHZENmNUMUlt?=
 =?utf-8?B?SlNlWXB1eFRDbGZacmxtMVhoYWxLNGNzSmpHaHZnNlg5VVozZU1ZbWRqUzlx?=
 =?utf-8?B?TDBKK3d4cmtQRnFkd2tOVnRWR2NPay9rc1d4VU85TGF3dUZ0azFJMENmc3pU?=
 =?utf-8?B?ZGNwSk9yMHFRSThReFVscWpPbzRGRzE5d2ZDN3VLcXJkc1hpMG9uT05EbG1z?=
 =?utf-8?B?enVsRUxJWXNjdHV5c3Q1WmViVUI4UjU4dURrMWcyeEkreTJRM2p1QWRIN0Ew?=
 =?utf-8?B?K3dINXp3a1FUVUVCYUM4c2p1RGVJbXN5NEgvRkpVQ0w3SzZ3UEFPQnNJMGVT?=
 =?utf-8?B?cXlTazVTY25LMldXelZROUxMTHFGNEhFR3UzL1ZhaFdraHJGTFYwUFlIdEFt?=
 =?utf-8?B?YTMzbzZtaCtqMkdBM0RqdkgwNXkyZUtZWWI2elFKS2VvcVV2REhaRDNER0NW?=
 =?utf-8?B?K09BbVJERG9zRVJhZ2cwNFY5TnNwa1FDdU5aeGo0aUlha0JwTjVZV0FnaXVj?=
 =?utf-8?B?NDh0a1B6TStpODJJc0ZpMDhNb1FOamhvTzVxemNHVUNzMnp5MVlXbXZVa2VV?=
 =?utf-8?B?ajZnWkcxc2QvUTllNFdSZlNDOXdPWnRvcFdwZy9ReENiOVMwODBxTWgySEtx?=
 =?utf-8?B?N1AvMUI5OUpkQ3Zvc1FEWUVGcE9zTGZjRFlRM1QveGZWY1gwajRIdXpkbzFq?=
 =?utf-8?B?SmN2Y0lCakVGcFRQTWt2Q2lWQWtmYXdZZzJSbUswWnpvUzdnK1pRMUEyaDF3?=
 =?utf-8?B?eEhWcWVIeEQ4ekkrdnROSjN5OVVzaHRvWFJWMHFvTEtBSE0rWGsyeXB3RGJG?=
 =?utf-8?B?T0d0NVBqbHJHSTllZVJLY2NuTlJyNDJ4OW55MDlYQmIreFJ1TjNqdGtwaGRE?=
 =?utf-8?B?UVBLZFdtMG8xcFUwbWJaQ25LVlFRRlFkZlppR1YvS3krYStYVlZzWTZrRnVn?=
 =?utf-8?B?TWdFOHZuSG5vODAxUjN6M2YvZThqbG1kSHQ5MVJKdFNSM3ZwUExURGE2T2pM?=
 =?utf-8?B?ZUpHM3FpOGZ2YXU0NkY5MERHR3JTNmNvRFUxVkxPUjFnaDZEWEhDTy8wUjlJ?=
 =?utf-8?B?Q0lOSjl2bEh1azU2VUYyTmxqVnJvRlJyWC9tOW9OYWpuRlVYeEFUT2RrOG5a?=
 =?utf-8?B?YnZLb0xjUjNtdURIK1RkN0NJNThVak1LNmxxK2xGVFhsbmV6b1BUSW5mOHdO?=
 =?utf-8?B?NDRmazJGUjNwTjZMZnRudXpkbnlGQzVZZ1FQaG8vcmdhNmIrQit1aDFYQnpj?=
 =?utf-8?B?OTFDaU1LSE5BRDdyZy9sT0hlY0VzNk1UY3pwVWEyZTZsb0pCOGY5dmlZeWtP?=
 =?utf-8?B?NjBTbTBmaVFWQVhHQUxObmJ3RFhZSERlUTBHYUZJbGNlZitBTkdScTk0NUR0?=
 =?utf-8?B?UGZTRzY3aDZrME1hdjg1TUZ4VEdncDZlRXhjN0RDYjBhV2lsUDNnLy9QbDAw?=
 =?utf-8?B?YmdqMjlQSUZvdUltZkNSSmQwZnl5aDBpeXVZdm9nQW4rUTlKZCt3V1lDS2Qv?=
 =?utf-8?B?Z1g1Nnc2SWVMMmFUUktsR3N6THdiZll5S2MrdExJUllPOTlEejR6MzZLc1BI?=
 =?utf-8?B?SjM1Q09ReUVMS3Y5R0ZxVlp5WkJoSHVNTVF2N3VWNHlmcC94NGFTL2YyVFRC?=
 =?utf-8?B?YWdKMXRmRTFnN0FOTnVqaUtYTWVNdU5id0NCQXcrZkp5Z09sdzdHZ0RSRndt?=
 =?utf-8?B?MkFOVWVXTnFtRVpvRzk1ejdhNm5vRzlrUWlROXU5a0tvNkhsSTd4eDhBM2lC?=
 =?utf-8?B?aVdwYWh6T1Z4ZHBZUGpUOUx4Nlh0WjJ4Z2VJOW5vUWhyc01UYmJReU1wQVhR?=
 =?utf-8?B?bjJwZ3gzVHFHYmpkZEE2aUl5NWZVWFR0TnlWcTFtdHNhbVRSQVBoRUpmZWJV?=
 =?utf-8?B?dU9IeFJrbTlFbUkreGNzNFY5akVkNndvT2FCNnJubEYzaVJMMGsybWRHcVRE?=
 =?utf-8?B?amRhbFc5alZEQlpNZnFMdk1XQXdzdnUwd25RMmlxUElGUXo4a2FiUWY3THFF?=
 =?utf-8?B?SCtmNTJqaHduK3FLY25OYnBBcmFUOEl4OTNFYjJKdE1Uc0lWd1BFOW9nemJz?=
 =?utf-8?Q?A2EClVOtBD8IV373xeo5Y+Kti?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 219f862b-7c7d-4e0a-7310-08dc5aa525f7
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2024 04:00:53.5751
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EoLWjSYqJcExoblJtz3zTuXiRAwc8ffnCdr2jGmA6NVVCyx4diICgLuQPACa56fjorndK0DD4PqeOEvvgyPBkQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7878


On 03-Apr-24 2:10 PM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
> 
>> On 02-Apr-24 7:33 AM, Huang, Ying wrote:
>>> Bharata B Rao <bharata@amd.com> writes:
>>>
>>>> On 29-Mar-24 6:44 AM, Huang, Ying wrote:
>>>>> Bharata B Rao <bharata@amd.com> writes:
>>>> <snip>
>>>>>> I don't think the pages are cold but rather the existing mechanism fails
>>>>>> to categorize them as hot. This is because the pages were scanned way
>>>>>> before the accesses start happening. When repeated accesses are made to
>>>>>> a chunk of memory that has been scanned a while back, none of those
>>>>>> accesses get classified as hot because the scan time is way behind
>>>>>> the current access time. That's the reason we are seeing the value
>>>>>> of latency ranging from 20s to 630s as shown above.
>>>>>
>>>>> If repeated accesses continue, the page will be identified as hot when
>>>>> it is scanned next time even if we don't expand the threshold range.  If
>>>>> the repeated accesses only last very short time, it makes little sense
>>>>> to identify the pages as hot.  Right?
>>>>
>>>> The total allocated memory here is 192G and the chunk size is 1G. Each
>>>> time one such 1G chunk is taken up randomly for generating memory accesses.
>>>> Within that 1G, 262144 random accesses are performed and 262144 such
>>>> accesses are repeated for 512 times. I thought that should be enough
>>>> to classify that chunk of memory as hot.
>>>
>>> IIUC, some pages are accessed in very short time (maybe within 1ms).
>>> This isn't repeated access in a long period.  I think that pages
>>> accessed repeatedly in a long period are good candidates for promoting.
>>> But pages accessed frequently in only very short time aren't.
>>
>> Here are the numbers for the 192nd chunk:
>>
>> Each iteration of 262144 random accesses takes around ~10ms
>> 512 such iterations are taking ~5s
>> numa_scan_seq is 16 when this chunk is accessed.
>> And no page promotions were done from this chunk. All the
>> time should_numa_migrate_memory() found the NUMA hint fault
>> latency to be higher than threshold.
>>
>> Are these time periods considered too short for the pages
>> to be detected as hot and promoted?
> 
> Yes.  I think so.  This is burst accessing, not repeated accessing.
> IIUC, NUMA balancing based promotion only works for repeated accessing
> for long time, for example, >100s.

Hmm... When a page is accessed 512 times over a period of 5s and it is
still not detected as hot. This is understandable if fresh scanning couldn't
be done as the accesses were bursty and hence they couldn't be captured via
NUMA hint faults. But here the access captured via hint fault is being rejected
as not hot because the scanning was done a while back. But I do see the challenge
here since we depend on scanning time to obtain the frequency-of-access metric.

BTW, for the above same scenario with numa_balancing_mode=1, the remote
accesses get detected and migration to source node is tried. It is a different
matter that eventually pages can't be migrated in this specific scenario as
the src node is already full.

Regards,
Bharata.