Received-SPF: pass (google.com: domain of linux-kernel+bounces-133370-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Fri, 5 Apr 2024 12:56:14 -0400
From: Gregory Price <gregory.price@memverge.com>
To: Honggyu Kim <honggyu.kim@sk.com>
Cc: sj@kernel.org, damon@lists.linux.dev, linux-mm@kvack.org,
	akpm@linux-foundation.org, apopple@nvidia.com,
	baolin.wang@linux.alibaba.com, dave.jiang@intel.com,
	hyeongtak.ji@sk.com, kernel_team@skhynix.com, linmiaohe@huawei.com,
	linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
	mathieu.desnoyers@efficios.com, mhiramat@kernel.org,
	rakie.kim@sk.com, rostedt@goodmis.org, surenb@google.com,
	yangx.jy@fujitsu.com, ying.huang@intel.com, ziy@nvidia.com,
	42.hyeyoo@gmail.com, art.jeongseob@gmail.com
Subject: Re: [RFC PATCH v3 0/7] DAMON based tiered memory management for CXL
 memory
Message-ID: <ZhAtLhcU3KfT/9i7@memverge.com>
References: <20240405060858.2818-1-honggyu.kim@sk.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240405060858.2818-1-honggyu.kim@sk.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?u3uA89lnbl1XjLpDljLpSx+EFbKrPa8e92AEIJA2eg+zB1ny2L3/gV8Qg9/v?=
 =?us-ascii?Q?wcmqhc2yi8DaWC9fvgDNxd6PkaR35SRxXdRAXuhy7VIUvuHrlQpX8JbtLbQa?=
 =?us-ascii?Q?WHL3K5gPSp3fkLUOXcj9IqqtvqkTJn0kgHWwxkX6DujT1C9/bLdd1alvLOIx?=
 =?us-ascii?Q?r9XUgWAsuCDxRG5O1ouee0zHz4YskyzcAlVRJFnq889ExUr3Nx7/C5Yg0SB6?=
 =?us-ascii?Q?wf84rLaaERa2af+b3FncGL1THgD+WBEUmn4Gx1wJ8SzSnP8/3uxIp+N5J+sV?=
 =?us-ascii?Q?f53Y/elCJNU4OEjg475PCsHZMMyNvVH/FDeHIlExXIruIGvUl61d30/v9y06?=
 =?us-ascii?Q?qcGDZCdnfLgduhHkhYbUD9CXkKaJFAI3QH2Yo77d6o/nrtLewQjlwsGJHFPj?=
 =?us-ascii?Q?czyzQKp62Keyn2kC8VtpSpTr7zRP2UbqgKH2lUcSHIldB/1PuhJUVjKc44I/?=
 =?us-ascii?Q?3nHPLwADQ7neG6Iism7HhhmJWHUnc5ci3myN2muWaPmPhn1ZpR9TR3GRNtRv?=
 =?us-ascii?Q?Rb9KKSqJd3HuzJye/R9tKHaGNGD+/gi+BkEKplMFpofkkB2ie5mmEUBzpFCc?=
 =?us-ascii?Q?K+0amEMcg/LE1ZnRrQA44sV4+MI8Ph9Ym7SKJ3AL+ZKnhVjOSe2woxBxITwo?=
 =?us-ascii?Q?fGLuyt/Z0y8hUPlRYQTQPo4ADWw9WphDEHg5VjbRkVMf5rtNehQnIDk3R6bB?=
 =?us-ascii?Q?ItRHxtdSyPTxZiRXuyu5k2G/m3cQS33+vnBcn54x9deeFP5VEjuG+mNJ9sxb?=
 =?us-ascii?Q?z0klIgIZTW0DEkjyXZO/QkKQLgjnOcvRMsAydG/0giVa+sJt9qgF7/rouNYu?=
 =?us-ascii?Q?ZA6DT3TAlYhZtd+0sNj+BFa6tqI7EXw6PDXiJDOtvaI7vjjd3hSA9Y7rTXjm?=
 =?us-ascii?Q?NC+8V1HedrbItuqTtFBdeXGK6vV5Mp8VKDXIMcGG+ccxNvsod/4WOkofgwMf?=
 =?us-ascii?Q?gBVJVVwpoJxzhVbldlHEtgfV7tFwZmNkeo1hBD3eXA7/NWTpaIbSCaE+zJMu?=
 =?us-ascii?Q?nLgzgQb2faSsniuXNVL5kcG5XDv8P7Yo29/LnL0196JU4GlhcxVV8LK9b6wK?=
 =?us-ascii?Q?3cQyzctryt1GNf7eXEzCyD+CyuI8FsfPF53Bn90yVtxD3AWA/HMSE58Bb7lx?=
 =?us-ascii?Q?Eshtol6LFJhI6CFV2Vxm1up/exZwxomRTsVomC/FUvtS7Gr5MUpH2kVwF56S?=
 =?us-ascii?Q?gGtS3ixv6VMKOgKASgKRFmLRd/PiLBRqH4/TZ575dlK3sAhtE6v8Of4/suet?=
 =?us-ascii?Q?1jGWsvOJiAW38wKkl3+FcWx/l63Y3+NI/cp7xndyjllelCztAEMcOkOknwVP?=
 =?us-ascii?Q?voAYGxwMXHashudX9asKuBZi4M/EMCY2pFWzwwFxi69orUuddx5u3E1PNVSz?=
 =?us-ascii?Q?WpUXJm7fm5pzQ4XbOPVShvxBUO3Cia1KCI90gUBbqfyqDxiE7WHvjFvVtM9i?=
 =?us-ascii?Q?46BNzP98PNJhlaYCQbgVaX7/22knhGPmEji6j53fPFfGjBPEkasMzwxq+Fgy?=
 =?us-ascii?Q?YbB8jFV3AFGjfjZSdHOY9+Yk2Fr7ptxOATJKeuXEtYOuqAb+og/UQSK9dN9+?=
 =?us-ascii?Q?TMHlreZk+aLwsOi75iSPYj41c/zlKgGgqG0Qd1jdUw59ADGBXADNqB5G7/qo?=
 =?us-ascii?Q?Jw=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6fbea0ab-734c-4fc0-54bc-08dc55915356
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2024 16:56:23.8254
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UeqfEUQg0/fVg0jo0o8bRoViY2vDLz8lEsCxvE92XDGxwp65zmDjTteUyTSaqlRis72353TqzOgym5ra8DFfwh3wIqbsbDKXIkbG+mTsOOM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB4582

On Fri, Apr 05, 2024 at 03:08:49PM +0900, Honggyu Kim wrote:
> There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously
> posted at [1].
> 
>   1. YCSB zipfian distribution read only workload
>   memory pressure with cold memory on node0 with 512GB of local DRAM.
>   =============+================================================+=========
>                |       cold memory occupied by mmap and memset  |
>                |   0G  440G  450G  460G  470G  480G  490G  500G |
>   =============+================================================+=========
>   Execution time normalized to DRAM-only values                 | GEOMEAN
>   -------------+------------------------------------------------+---------
>   DRAM-only    | 1.00     -     -     -     -     -     -     - | 1.00
>   CXL-only     | 1.22     -     -     -     -     -     -     - | 1.22
>   default      |    -  1.12  1.13  1.14  1.16  1.19  1.21  1.21 | 1.17 
>   DAMON tiered |    -  1.04  1.03  1.04  1.06  1.05  1.05  1.05 | 1.05 
>   =============+================================================+=========
>   CXL usage of redis-server in GB                               | AVERAGE
>   -------------+------------------------------------------------+---------
>   DRAM-only    |  0.0     -     -     -     -     -     -     - |  0.0
>   CXL-only     | 52.6     -     -     -     -     -     -     - | 52.6
>   default      |    -  20.4  27.0  33.1  39.5  45.6  50.5  50.3 | 38.1
>   DAMON tiered |    -   0.1   0.3   0.8   0.6   0.7   1.3   0.9 |  0.7
>   =============+================================================+=========
> 
> Each test result is based on the exeuction environment as follows.
> 
>   DRAM-only   : redis-server uses only local DRAM memory.
>   CXL-only    : redis-server uses only CXL memory.
>   default     : default memory policy(MPOL_DEFAULT).
>                 numa balancing disabled.
>   DAMON tiered: DAMON enabled with DAMOS_MIGRATE_COLD for DRAM nodes and
>                 DAMOS_MIGRATE_HOT for CXL nodes.
> 
> The above result shows the "default" execution time goes up as the size
> of cold memory is increased from 440G to 500G because the more cold
> memory used, the more CXL memory is used for the target redis workload
> and this makes the execution time increase.
> 
> However, "DAMON tiered" result shows less slowdown because the
> DAMOS_MIGRATE_COLD action at DRAM node proactively demotes pre-allocated
> cold memory to CXL node and this free space at DRAM increases more
> chance to allocate hot or warm pages of redis-server to fast DRAM node.
> Moreover, DAMOS_MIGRATE_HOT action at CXL node also promotes hot pages
> of redis-server to DRAM node actively.
> 
> As a result, it makes more memory of redis-server stay in DRAM node
> compared to "default" memory policy and this makes the performance
> improvement.
> 
> The following result of latest distribution workload shows similar data.
> 
>   2. YCSB latest distribution read only workload
>   memory pressure with cold memory on node0 with 512GB of local DRAM.
>   =============+================================================+=========
>                |       cold memory occupied by mmap and memset  |
>                |   0G  440G  450G  460G  470G  480G  490G  500G |
>   =============+================================================+=========
>   Execution time normalized to DRAM-only values                 | GEOMEAN
>   -------------+------------------------------------------------+---------
>   DRAM-only    | 1.00     -     -     -     -     -     -     - | 1.00
>   CXL-only     | 1.18     -     -     -     -     -     -     - | 1.18
>   default      |    -  1.18  1.19  1.18  1.18  1.17  1.19  1.18 | 1.18 
>   DAMON tiered |    -  1.04  1.04  1.04  1.05  1.04  1.05  1.05 | 1.04 
>   =============+================================================+=========
>   CXL usage of redis-server in GB                               | AVERAGE
>   -------------+------------------------------------------------+---------
>   DRAM-only    |  0.0     -     -     -     -     -     -     - |  0.0
>   CXL-only     | 52.6     -     -     -     -     -     -     - | 52.6
>   default      |    -  20.5  27.1  33.2  39.5  45.5  50.4  50.5 | 38.1
>   DAMON tiered |    -   0.2   0.4   0.7   1.6   1.2   1.1   3.4 |  1.2
>   =============+================================================+=========
> 
> In summary of both results, our evaluation shows that "DAMON tiered"
> memory management reduces the performance slowdown compared to the
> "default" memory policy from 17~18% to 4~5% when the system runs with
> high memory pressure on its fast tier DRAM nodes.
> 
> Having these DAMOS_MIGRATE_HOT and DAMOS_MIGRATE_COLD actions can make
> tiered memory systems run more efficiently under high memory pressures.
> 

Hi,

It's hard to determine from your results whether the performance
mitigation is being caused primarily by MIGRATE_COLD freeing up space
for new allocations, or from some combination of HOT/COLD actions
occurring during execution but after the database has already been
warmed up.

Do you have test results which enable only DAMOS_MIGRATE_COLD actions
but not DAMOS_MIGRATE_HOT actions? (and vice versa)

The question I have is exactly how often is MIGRATE_HOT actually being
utilized, and how much data is being moved. Testing MIGRATE_COLD only
would at least give a rough approximation of that.


Additionally, do you have any data on workloads that exceed the capacity
of the DRAM tier?  Here you say you have 512GB of local DRAM, but only
test a workload that caps out at 500G.  Have you run a test of, say,
550GB to see the effect of DAMON HOT/COLD migration actions when DRAM
capacity is exceeded?

Can you also provide the DRAM-only results for each test?  Presumably,
as workload size increases from 440G to 500G, the system probably starts
using some amount of swap/zswap/whatever.  It would be good to know how
this system compares to swap small amounts of overflow.

~Gregory