Received-SPF: pass (google.com: domain of linux-kernel+bounces-126645-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com>
Date: Mon, 1 Apr 2024 17:50:57 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address
 space
Content-Language: en-US
To: "Huang, Ying" <ying.huang@intel.com>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
 akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org,
 mgorman@techsingularity.net, raghavendra.kt@amd.com,
 dave.hansen@linux.intel.com, hannes@cmpxchg.org
References: <20240327160237.2355-1-bharata@amd.com>
 <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <dd2bc563-7654-4d83-896e-49a7291dd1aa@amd.com>
 <87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com>
 <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com>
From: Bharata B Rao <bharata@amd.com>
In-Reply-To: <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?cmhyRitFeGIrNHpycFRHanowZTBOTzlLTXBjTEZmZ1l5Qmc2U1gyYW0vekNS?=
 =?utf-8?B?b3YrWmJzTU9SbHVnVTI5dEtmQ1lLZ0hHUTJkMGV3dlc5KzVlQW84eHNQL2c5?=
 =?utf-8?B?VUsvSWN2WjdZVXg2OTdRejEvZHErc1ZPOWMyTDQ4aGRWWjFxSjJXdmIrVXkv?=
 =?utf-8?B?enBobFRKM2JBaVRDa1ZIZVIxN3VCNmV5TW1Za01jemVwVGtZV0M0Z21nemxr?=
 =?utf-8?B?TjFkdGJPZmJLQlhZenVaSE9PTENxVWxJZkVHWk9Mek1XRGhneHh2WVkzajg2?=
 =?utf-8?B?TVQ1MHMrUEczSFlWS2RuRVlmRXhqZUZXVzZIS2crVnFHZnJlNG9reDg3VWhN?=
 =?utf-8?B?cjVFajI3ZzJyejJibXJzYnl3K3c1ZUJaUFBXaUw4ZFdyWFpTS2RwTDR6M0M3?=
 =?utf-8?B?clJyWXBuVnp5VVJoY0gyV09YdCtIek9JWjRoRUtqZFhQZkFTREVSc2VabU1L?=
 =?utf-8?B?bVVwY3cveTJCaTBRNEZRcU1mTDZ2bnlGb29lVTdqaDYyWHR4dXZRM2FxbFhI?=
 =?utf-8?B?WmpNR0YrVFkyUWtHWTV1cmcxWWdhem9DbHE1SjJEQktOZUVXRTh5ZndXUjNB?=
 =?utf-8?B?UVI4UU9ySGNlNjdUWFo0QWdDN0JWNktHQ0xHTFlyYVpXOUFUbFNCUzlTKzZv?=
 =?utf-8?B?YkMwV1ZMSUNXbG43WVhSbm9kbWY0MEg5RVJ2RGZuVmp1UGZTVmUvbW1OekZk?=
 =?utf-8?B?Q3NGR09Ka05SQ01RWWtubTZnUlBMekxvZ2hwMTk4eSs0OFlTVlRkZGl2Njhz?=
 =?utf-8?B?TDJHZnBOZ05rVVZnRWpReEVwTEE5UTBRU0tPWklOaEJmbDh6OGJ1QnZ2Z05S?=
 =?utf-8?B?UXR3c2pKUUZDV1dVRXdyTzkxN0wwZ3RCR2x4TkJ0eHBvTzdyeTlUQmRrY0ta?=
 =?utf-8?B?ZmdxYmFLTkR1a3ZFYXZwYlBNTzgvdkFpcVkzUWRkdFlyMEtQYWFmMExndDRO?=
 =?utf-8?B?NlV4RDZwaHVuZnRGaU1QT2JDWndIMlRUVVlQVC91aEpaaUs1ei9PN2dwcWtz?=
 =?utf-8?B?RVJvdXRnYXZ5Y0g0QmcxQ20wSWRodTRVem1lYkRGYjErc1JxVndvcG16a2hM?=
 =?utf-8?B?N3VTcC9EOWN1R0VyNUVtdmh2eklqYmlaS2MwL3BRSU1NeEZZaGZ4dE5GMGdC?=
 =?utf-8?B?NEJ0ejR0S2ZQRVF5RWR5dDdTUzJML1VSTStHeGFkQ1NaL0RQN2JMSGRBYWlD?=
 =?utf-8?B?bzZYb3lMc2E3UGU5YU0vdEQ0K2NyNFdpTkloT00xMEE2V2hPNHQxZ3F1bTlv?=
 =?utf-8?B?QTZCQ0hMd2NnOFJkMkRFZTlXcFZJNzduaGczR0RZUDkxNGh1ZC8vYm1rNU9T?=
 =?utf-8?B?NEJIY1dFK1cyQmlJQ3JIaHBlWGZISFNuczFkN05uN1ZpNEdtNnFPNnpjZ2Vn?=
 =?utf-8?B?YVRlMHd1LzNvQnUyMzVmbHVuVm03RklDQmFzME5KZHRjY0NOQzlrWDFXZUJn?=
 =?utf-8?B?a3B3YWV0K0luOEtFV1JOQSs4dHBWbmpQRE8yWlZLYStLSDY5dUhxdzJVTGFl?=
 =?utf-8?B?b1p5MkovdTMzdnJnYTErNWZUOE5qWENaL3lhcVlDZFVNNXBhTGJFbWc0c0tl?=
 =?utf-8?B?VVpkVEtlQWx2NFBnQXlZQnJKY29FSFVzNFZyeWdRUHM4OUJtQVlUcndzVCt4?=
 =?utf-8?B?dGNLWVMyQ1hFdCtMOVVJR1ZMc292eWVGRDI2ZWpSamEyUytFazVzSG5xd29o?=
 =?utf-8?B?T3hMWEdPYSsyRnhFcVNOTE1KY05iMlF6RlZVWmZIRWJpSUlKNnNYVDNySTdX?=
 =?utf-8?B?NDJaaHRJTm12bWQ1bVFUQS9kYTh6bnkvb1Z1OG50bFpyMjdYS2t3SnIxWk91?=
 =?utf-8?B?L0JaNk1OYllpNFB5d3p3ZE9PZEU5ZGhDUnhKK240cXZqUFhYTXBFSk0vOEFu?=
 =?utf-8?B?OVJ0cGt3WkwzTjJJbzF1LzB3Y3JoQ2lWR0RhZXJvRzBGN2Iwbk1VVDIweVJE?=
 =?utf-8?B?dzFVc3lHYmVMRDUySHdpMVg0OGhqZC84ekxyVlFjWGtYclFvaXJhVWNEdFNo?=
 =?utf-8?B?a3RQekJlWWNXdWQvSGsyTjdUb0NwUHJBOWJyUWJqMExwK1FjNTl3cDJLUkl6?=
 =?utf-8?B?VGdjazVzTWFyblgvMG5xc1F6VjYrVlRrdTRpZEpxM1ZwR1VHUENLZmtBSlFI?=
 =?utf-8?Q?Xg3X/dNDyGRHsT0epIqyV2/7t?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 658b69db-c9c3-4b96-6083-08dc524633f2
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Apr 2024 12:21:05.5658
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ELpej1ZvPlZ8V+kfjdp4shUVu3lkv3PZ8S05LZ0XNvSJ+jsooyNPUCCRsn6s0zJ7kg6kZYAVeAQNsR9yoPkihg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5900

On 29-Mar-24 6:44 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
<snip>
>> I don't think the pages are cold but rather the existing mechanism fails
>> to categorize them as hot. This is because the pages were scanned way
>> before the accesses start happening. When repeated accesses are made to
>> a chunk of memory that has been scanned a while back, none of those
>> accesses get classified as hot because the scan time is way behind
>> the current access time. That's the reason we are seeing the value
>> of latency ranging from 20s to 630s as shown above.
> 
> If repeated accesses continue, the page will be identified as hot when
> it is scanned next time even if we don't expand the threshold range.  If
> the repeated accesses only last very short time, it makes little sense
> to identify the pages as hot.  Right?

The total allocated memory here is 192G and the chunk size is 1G. Each
time one such 1G chunk is taken up randomly for generating memory accesses.
Within that 1G, 262144 random accesses are performed and 262144 such
accesses are repeated for 512 times. I thought that should be enough
to classify that chunk of memory as hot. But as we see, often times
the scan time is lagging the access time by a large value.

Let me instrument the code further to learn more insights (if possible)
about the scanning/fault time behaviors here.

Leaving the fault count based threshold apart, do you think there is
value in updating the scan time for skipped pages/PTEs during every
scan so that the scan time remains current for all the pages?

> 
> The bits to record scan time or hint page fault is limited, so it's
> possible for it to overflow anyway.  We scan scale time stamp if
> necessary (for example, from 1ms to 10ms).  But it's hard to scale fault
> counter.  And nobody can guarantee the frequency of hint page fault must
> be less 1/ms, if it's 10/ms, it can record even short interval.

Yes, with the approach I have taken, the time factor is out of the
equation and the notion of hotness is purely a factor of the number of
faults (or accesses)

Regards,
Bharata.