Received-SPF: pass (google.com: domain of linux-kernel+bounces-122430-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: "Huang, Ying" <ying.huang@intel.com>
To: Bharata B Rao <bharata@amd.com>
Cc: <linux-mm@kvack.org>,  <linux-kernel@vger.kernel.org>,
  <akpm@linux-foundation.org>,  <mingo@redhat.com>,
  <peterz@infradead.org>,  <mgorman@techsingularity.net>,
  <raghavendra.kt@amd.com>,  <dave.hansen@linux.intel.com>,
  <hannes@cmpxchg.org>
Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large
 address space
In-Reply-To: <20240327160237.2355-1-bharata@amd.com> (Bharata B. Rao's message
	of "Wed, 27 Mar 2024 21:32:35 +0530")
References: <20240327160237.2355-1-bharata@amd.com>
Date: Thu, 28 Mar 2024 13:35:42 +0800
Message-ID: <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13)
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii

Bharata B Rao <bharata@amd.com> writes:

> In order to check how efficiently the existing NUMA balancing
> based hot page promotion mechanism can detect hot regions and
> promote pages for workloads with large memory footprints, I
> wrote and tested a program that allocates huge amount of
> memory but routinely touches only small parts of it.
>
> This microbenchmark provisions memory both on DRAM node and CXL node.
> It then divides the entire allocated memory into chunks of smaller
> size and randomly choses a chunk for generating memory accesses.
> Each chunk is then accessed for a fixed number of iterations to
> create the notion of hotness. Within each chunk, the individual
> pages at 4K granularity are again accessed in random fashion.
>
> When a chunk is taken up for access in this manner, its pages
> can either be residing on DRAM or CXL. In the latter case, the NUMA
> balancing driven hot page promotion logic is expected to detect and
> promote the hot pages that reside on CXL.
>
> The experiment was conducted on a 2P AMD Bergamo system that has
> CXL as the 3rd node.
>
> $ numactl -H
> available: 3 nodes (0-2)
> node 0 cpus: 0-127,256-383
> node 0 size: 128054 MB
> node 1 cpus: 128-255,384-511
> node 1 size: 128880 MB
> node 2 cpus:
> node 2 size: 129024 MB
> node distances:
> node   0   1   2 
>   0:  10  32  60 
>   1:  32  10  50 
>   2:  255  255  10
>
> It is seen that number of pages that get promoted is really low and
> the reason for it happens to be that the NUMA hint fault latency turns
> out to be much higher than the hot threshold most of the times. Here
> are a few latency and threshold sample values captured from
> should_numa_migrate_memory() routine when the benchmark was run:
>
> latency	threshold (in ms)
> 20620	1125
> 56185	1125
> 98710	1250
> 148871	1375
> 182891	1625
> 369415	1875
> 630745	2000

The access latency of your workload is 20s to 630s, which appears too
long.  Can you try to increase the range of threshold to deal with that?
For example,

echo 100000 > /sys/kernel/debug/sched/numa_balancing/hot_threshold_ms

[snip]

--
Best Regards,
Huang, Ying