Received-SPF: pass (google.com: domain of linux-kernel+bounces-127631-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <9ec3b04b-bde8-42ce-be1b-34f7d8e6762d@amd.com>
Date: Tue, 2 Apr 2024 14:56:37 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address
 space
Content-Language: en-US
To: "Huang, Ying" <ying.huang@intel.com>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
 akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org,
 mgorman@techsingularity.net, raghavendra.kt@amd.com,
 dave.hansen@linux.intel.com, hannes@cmpxchg.org
References: <20240327160237.2355-1-bharata@amd.com>
 <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <dd2bc563-7654-4d83-896e-49a7291dd1aa@amd.com>
 <87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com>
 <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com>
 <87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com>
From: Bharata B Rao <bharata@amd.com>
In-Reply-To: <87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ejQzWmdrTTlkRmVxQ1REcmJmeHkzR29TajEremprMGtONlhUZ2Q1YUw0ekx5?=
 =?utf-8?B?eVdNTW9LU0x2Ylg2a2M2Nml3WWtHc3JFV1FvaEF1bXFSS1RuN1VPSG44dWVo?=
 =?utf-8?B?VTA0OXRuQUUrM21zOGxDMm5obmFPcGxibHo1a3pDaTZSYy9SbGFNcElkeDZw?=
 =?utf-8?B?ZlR2eHVtZFFlWjBLa0dXVE5ielR3azNTbDJESHhWcFRhNWhGMk1xaVFDY1d1?=
 =?utf-8?B?ZGhNZEk3ZDcyZldjbjkwbm9uaWtZZVpWNTVoOTJPbmMrSUowTW1NUXRmZ0l0?=
 =?utf-8?B?QlF0VmovKy9MRlJZQWlZcEROKzN3OENTUVh2bzRGRmQ1aE80RndLNFpOR2lY?=
 =?utf-8?B?ZmlkZ2J5WXVRL0FKNHZmL0ZZL052b29iSE1xRmZRNS9MZFNhcDZTM2hkNHRu?=
 =?utf-8?B?RXU0TFd3eHZwUG9TK0lHUUlrajVkbEV5TXdaaG5YVXBKVmtWTWhoV011U3lT?=
 =?utf-8?B?K2hmMnk1SzlMaEE3UE1PeG1iZ2ZDUGIrc1BzaENTSitRamVjZzQzQjJVMTk2?=
 =?utf-8?B?YXdERVVoTzdtSldLSlluM2FPZXNxYklrcHUxNFg2WWQ1Tk95MmF6Y0RKTzM4?=
 =?utf-8?B?L1dIVytCR0JQa1kwL2hGck5VQlVXeUZ6MXU5RU1vTVovT1lwcXUrQXpwcWR6?=
 =?utf-8?B?aW9ER09nY2wyeW5nOHJNWHc5RVZ1UlFtbncxMldSbUM5YjQ1ZVdkajlRQmpX?=
 =?utf-8?B?MFFPMGh3dzR3MDFSTHZxMVpOTk5mWHg5RWkvem9pS2Z1c1FMQkR1dVQzVjdI?=
 =?utf-8?B?R3NVa010clN2aXZQSXJlK1RVd0hYdmZXa1grbWJheGZ6TktwRmFpMS9WbE1a?=
 =?utf-8?B?am5DbElWMXY5WlFaVjEvUHR0WENxS2oyL0ZsMmd1dklZdHF5MjRMc2EyaEdn?=
 =?utf-8?B?RGNzaDFZeWlCdEpPd0Y1VjdPTFp2NjZ4b3dPakh2eXl5aGQ1YTB4ZnV6N05X?=
 =?utf-8?B?SGJybFkvNHhaSWtJR2svcVV1aGNMOS9FMUJoaXN5NVpUYmNGUkZ3UDBSbk9u?=
 =?utf-8?B?WHhNb2draHZFMGRaazdUblFWUUNXbkJQY2pLN3E0WlJJc0JYUjBQL2dzSmI0?=
 =?utf-8?B?VVBCSnJKNlVnYkxqVXVXeS9yRlUrajUvcnFkMWxJRkE1ZDZ6T3JWdVU4NjZB?=
 =?utf-8?B?SmM5VXVCSmVhNmdVWXZ5ZTFnZHVHL011amRXM1hsQkliTjFmKzd1eHE1Z2Iy?=
 =?utf-8?B?ekxzNGdUMGNVVGF1bXBqQ1hMOVdWaFB6aHVaYWo3T1MxaXVjQnhCU2EwK1VY?=
 =?utf-8?B?bzFORUFNSW5PeEdlT01NNG1lbldUZFFKSnNjVnVLVmtSY0dJWDRCVWRlNDdx?=
 =?utf-8?B?S1l2QU55amt0K0xvNlZIUGQrejVpSjRpMXZUQ2VaZXJDSlp2ZkVLUWxSQWtT?=
 =?utf-8?B?dkNkKzJDdlI1d2xQVVpXR09BOHVOSng2Vmk3a0VXUkpKcll1NEg1NFVEdERr?=
 =?utf-8?B?cTVtdGI4SWc4UklSODd3VzhBa1oyRmtacU82ellYT0tjajQ0Z3NKbHNRTnlD?=
 =?utf-8?B?OXRyVjRYU3VKUVlkeUNLR3lYL2djTmNnbkRDaXBTRTZpdEFFRUlpRkRsL0RB?=
 =?utf-8?B?bFo3bWxRNFpGYXpyS2NQdnhGQWJkRzdYTVRKS3ZWams2cEszR3A4eWViYlhv?=
 =?utf-8?B?NmRBcjkyeTdkV3RQKzNVNnA3U1VJQVNoYXZkdFQ0WUEreUczbDNnUS9jWndX?=
 =?utf-8?B?YWRBN3VobE95SThBTlJSSXMrb01kT3RMSG5hZlo3Uk9uMXBwaTc4QWFHTjNR?=
 =?utf-8?B?YWd4S0ZvNVMrdmtlN2RZZFFPNmJtcVFhb0V3RHh0RE4vSWEwMzBRUWdNSjRT?=
 =?utf-8?B?YVR2RGl1MTJidloyakU4VXZmSEJGSzN1TUo5WklkVTFFZC9Nd1VvK3Nmd2R6?=
 =?utf-8?B?eTdSMTBObWQ5SFlSaDFLWHpNV3ZWazRyYlRvMHFoajlaWFhUc1NHSldNdytt?=
 =?utf-8?B?VmFoWnVIbmNtTmVkWEtxdlJESVBmdFp0bnFkTnoxWGxWWnVaOTF5b1RKV2gz?=
 =?utf-8?B?bUc0YmNqVEwzUW1JUG0wWGpxYUV6UmVRTWg5d3F5SHlTZ3RMVHVKcFQ4Unkv?=
 =?utf-8?B?c2J0UFFockNwVWZYZUI0Qmp2QTRXNkE4UlMvdWE5M1RnMG1jd2tFMFVjekov?=
 =?utf-8?Q?K4iyDritK1ch4W70B5y5xGWBW?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b78bdef7-049e-4c6d-a6cf-08dc52f703db
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Apr 2024 09:26:45.7830
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FkQV4IrV0+TayFdOLIur+4Ru0EEPIASQbjP5MbYArn0jbWcISfNoNAcMsEkW9GWOmoYxD1NzhYA2Gfk8Xmnneg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8171

On 02-Apr-24 7:33 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
> 
>> On 29-Mar-24 6:44 AM, Huang, Ying wrote:
>>> Bharata B Rao <bharata@amd.com> writes:
>> <snip>
>>>> I don't think the pages are cold but rather the existing mechanism fails
>>>> to categorize them as hot. This is because the pages were scanned way
>>>> before the accesses start happening. When repeated accesses are made to
>>>> a chunk of memory that has been scanned a while back, none of those
>>>> accesses get classified as hot because the scan time is way behind
>>>> the current access time. That's the reason we are seeing the value
>>>> of latency ranging from 20s to 630s as shown above.
>>>
>>> If repeated accesses continue, the page will be identified as hot when
>>> it is scanned next time even if we don't expand the threshold range.  If
>>> the repeated accesses only last very short time, it makes little sense
>>> to identify the pages as hot.  Right?
>>
>> The total allocated memory here is 192G and the chunk size is 1G. Each
>> time one such 1G chunk is taken up randomly for generating memory accesses.
>> Within that 1G, 262144 random accesses are performed and 262144 such
>> accesses are repeated for 512 times. I thought that should be enough
>> to classify that chunk of memory as hot.
> 
> IIUC, some pages are accessed in very short time (maybe within 1ms).
> This isn't repeated access in a long period.  I think that pages
> accessed repeatedly in a long period are good candidates for promoting.
> But pages accessed frequently in only very short time aren't.

Here are the numbers for the 192nd chunk:

Each iteration of 262144 random accesses takes around ~10ms
512 such iterations are taking ~5s
numa_scan_seq is 16 when this chunk is accessed.
And no page promotions were done from this chunk. All the
time should_numa_migrate_memory() found the NUMA hint fault
latency to be higher than threshold.

Are these time periods considered too short for the pages
to be detected as hot and promoted?

> 
>> But as we see, often times
>> the scan time is lagging the access time by a large value.
>>
>> Let me instrument the code further to learn more insights (if possible)
>> about the scanning/fault time behaviors here.
>>
>> Leaving the fault count based threshold apart, do you think there is
>> value in updating the scan time for skipped pages/PTEs during every
>> scan so that the scan time remains current for all the pages?
> 
> No, I don't think so.  That makes hint page fault latency more
> inaccurate.

For the case that I have shown, depending on a old value of scan
time doesn't work well when pages get accessed after a long time
since scanning. At least with the scheme I show in patch 2/2,
probability of detecting pages as hot increases.

Regards,
Bharata.