Received-SPF: pass (google.com: domain of linux-kernel+bounces-137741-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Tue, 9 Apr 2024 20:00:18 -0400
From: Gregory Price <gregory.price@memverge.com>
To: Honggyu Kim <honggyu.kim@sk.com>
Cc: sj@kernel.org, damon@lists.linux.dev, linux-mm@kvack.org,
	akpm@linux-foundation.org, apopple@nvidia.com,
	baolin.wang@linux.alibaba.com, dave.jiang@intel.com,
	hyeongtak.ji@sk.com, kernel_team@skhynix.com, linmiaohe@huawei.com,
	linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
	mathieu.desnoyers@efficios.com, mhiramat@kernel.org,
	rakie.kim@sk.com, rostedt@goodmis.org, surenb@google.com,
	yangx.jy@fujitsu.com, ying.huang@intel.com, ziy@nvidia.com,
	42.hyeyoo@gmail.com, art.jeongseob@gmail.com
Subject: Re: [RFC PATCH v3 0/7] DAMON based tiered memory management for CXL
Message-ID: <ZhXWksusMnVKElmf@memverge.com>
References: <ZhAtLhcU3KfT/9i7@memverge.com>
 <20240408134108.2970-1-honggyu.kim@sk.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240408134108.2970-1-honggyu.kim@sk.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?US1L8cdvj4tgQXoBj5jjzmDtby/BO7BeIrJJWLeFMR4gCujotsGWJ5T+pMP9?=
 =?us-ascii?Q?HalZtx13aLwCKUk5blE7Jm2wVFePxClMjaIlWMDtqaf9c2+12ipFU4lhOQ3G?=
 =?us-ascii?Q?yj9dXc/ZlXYEUNBKm+HdMvei3Jmv/EbM8dGv6RO2+AJOAxwKSSB1lmh9qJOd?=
 =?us-ascii?Q?RoSndDpmKFm2TucDxHHEknHwbennuAo1qlssSk1B9RbCE6BeyeM9ljU/jsCm?=
 =?us-ascii?Q?8uBOTh/AaO54ZHqSgAjhdjFW78LHVsyT6GmmmV4KPHVT4NOWru6ZMPgo3/kv?=
 =?us-ascii?Q?8jKNZccsPJq99RjqG170gJhf1HHJFqSm+SFvWwonD4xAOrIjTSLvI7cS66NW?=
 =?us-ascii?Q?8HmVbdI8al9MNWBgd38nHRWF7SLaDRuDNaXrmLhsY+ZHOBmss0sXZdeaOO3q?=
 =?us-ascii?Q?SVa7vofXAX4KxiFW8EF4vdywEEr83XmYLT5PnQCDeiynwp0ZmJ5EkPNlKc2I?=
 =?us-ascii?Q?xaPGdwG31VWeShnEscy4vCxuHoMAzoBp3RhtcW8U5N51vTOdWGn0UHMXejDd?=
 =?us-ascii?Q?VqorpkSBAnSJW971cB2vtS5Fx2F4QfJCaETeBXklnCgy9QZu5YL5MZrW95Y+?=
 =?us-ascii?Q?+P//BdaUUyfMbNbVKiS4cbCiGevFNzYPQN6pykVEm7NvHmt4TErq3Q4gSeom?=
 =?us-ascii?Q?JuiIk8DVZB1PDGead7ydBMjrQW0Dp/kVYXaXt5l70UQjBe5ZlhrEDyOYqFIx?=
 =?us-ascii?Q?fl+WM2/cHtlqG7XTDM5C7FnJ/0VngOxGDjPDMMEPiL9HcB0sAByRtbTvnu0y?=
 =?us-ascii?Q?GIJrPn2QyivYx90K/2t/AJhMKog6ed7luNou21wgakEISBQ2lgnKNm9YteUK?=
 =?us-ascii?Q?deJWoudBsmb+lmWS3FKOVfwfW/RgTQfhvGbtcrzP5pJSn+Y2lJSTbERTonXN?=
 =?us-ascii?Q?t/94Fd/Bh0yGaY4FYwElgYiVWLIB9Hs7Y+EegKG6RD1n4PRYHiQTqC7I3wo3?=
 =?us-ascii?Q?tHKCEg/+QYiHSLH2b16ROspkYy7HsCIyxmZ7eoPi6IUhGvYJwobD5y70aLAg?=
 =?us-ascii?Q?qZMw8kCjVK1aJdm8W/OX/Yb8Qx7d9RrQ3a5dAO0NbMUqK/9LH6PckSaBqLI5?=
 =?us-ascii?Q?a7aZO9ynUdf5nexscAys+SePFHS+ygpK/qa+UYPmeE5i42nlsFcEBuawDvst?=
 =?us-ascii?Q?JQlqWF7e4gD88nMJQO+kasH/AWmsK1F3EpCnqA4ohv2CEfzAnWRTG5dNmz/l?=
 =?us-ascii?Q?uS+Y6sBf5pQP4Fify3aMfwtD6vgKscoNwcfPskfizj3vC4VhPBeQIBZCZvm9?=
 =?us-ascii?Q?g1cnmFy0hvJ1ScZfKInKUtJ4VuNdmdkMmC/bTDKzSulJnbRod7Q7Vis7Y0xX?=
 =?us-ascii?Q?sGoQFAUHAiROuohNlhlEO30jyq4jtMZdEl4gOJpUEnNij8wPyrlK7sTjx0Aq?=
 =?us-ascii?Q?2o7R5dEdzRyiNc2pjAgUA4wyobULMl6h6nKl787eyyhNuW0E0+/MO1zNaOCP?=
 =?us-ascii?Q?yzlKTTAum1qqJ8GnGVCLFp0iz3Vu4Rzoy8ohuVuEzz34uI+uEempvtJjpt9C?=
 =?us-ascii?Q?iAOYRvWYcNCMxp/ljCeY7sFiC/+2gyHPC+olHKsyKYcwX8bK7H2HaxUzLKiP?=
 =?us-ascii?Q?p+wPGSaI+0fmcbdllYzesScxo5hkziPZVlaytVy1Z3arIV9t6HUOEL1Iah6d?=
 =?us-ascii?Q?MQ=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2443ee24-29be-4c1c-65c6-08dc58f13994
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 00:00:25.7272
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IOkzgLdo0JO4LXoLigd1K4GJhsOYURC1YqbN9VlUkt42vklrlrxLv7ZKVrfRQfyT3xzMb3W5z5GAt7DHBgwUGrK0ikPlxVw8PsNGVsF6mO0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB4398

On Mon, Apr 08, 2024 at 10:41:04PM +0900, Honggyu Kim wrote:
> Hi Gregory,
> 
> On Fri, 5 Apr 2024 12:56:14 -0400 Gregory Price <gregory.price@memverge.com> wrote:
> > Do you have test results which enable only DAMOS_MIGRATE_COLD actions
> > but not DAMOS_MIGRATE_HOT actions? (and vice versa)
> > 
> > The question I have is exactly how often is MIGRATE_HOT actually being
> > utilized, and how much data is being moved. Testing MIGRATE_COLD only
> > would at least give a rough approximation of that.
> 
> To explain this, I better share more test results.  In the section of
> "Evaluation Workload", the test sequence can be summarized as follows.
> 
>   *. "Turn on DAMON."
>   1. Allocate cold memory(mmap+memset) at DRAM node, then make the
>      process sleep.
>   2. Launch redis-server and load prebaked snapshot image, dump.rdb.
>      (85GB consumed: 52GB for anon and 33GB for file cache)

Aha! I see now, you are allocating memory to ensure the real workload
(redis-server) pressures the DRAM tier and causes "spillage" to the CXL
tier, and then measure the overhead in different scenarios.

I would still love to know what the result of a demote-only system would
produce, mosty because it would very clearly demonstrate the value of
the demote+promote system when the system is memory-pressured.

Given the additional results below, it shows a demote-only system would
likely trend toward CXL-only, and so this shows an affirmative support
for the promotion logic.

Just another datum that is useful and paints a more complete picture.

> I didn't want to make the evaluation too long in the cover letter, but
> I have also evaluated another senario, which lazyly enabled DAMON just
> before YCSB run at step 4.  I will call this test as "DAMON lazy".  This
> is missing part from the cover letter.
> 
>   1. Allocate cold memory(mmap+memset) at DRAM node, then make the
>      process sleep.
>   2. Launch redis-server and load prebaked snapshot image, dump.rdb.
>      (85GB consumed: 52GB for anon and 33GB for file cache)
>   *. "Turn on DAMON."
> 
> In the "DAMON lazy" senario, DAMON started monitoring late so the
> initial redis-server placement is same as "default", but started to
> demote cold data and promote redis data just before YCSB run.
>

This is excellent and definitely demonstrates part of the picture I was
alluding to, thank you for the additional data!

> 
> I have included "DAMON lazy" result and also the migration size by new
> DAMOS migrate actions.  Please note that demotion size is way higher
> than promotion because promotion target is only for redis data, but
> demotion target includes huge cold memory allocated by mmap + memset.
> (there could be some ping-pong issue though.)
> 
> As you mentioned, "DAMON tiered" case gets more benefit because new
> redis allocations go to DRAM more than "default", but it also gets
> benefit from promotion when it is under higher memory pressure as shown
> in 490G and 500G cases.  It promotes 22GB and 17GB of redis data to DRAM
> from CXL.

I think a better way of saying this is that "DAMON tiered" more
effectively mitigates the effect of memory-pressure on faster tier
before spillage occurs, while "DAMON lazy" demonstrates the expected
performance of the system after memory pressure outruns the demotion
logic, so you wind up with hot data stuck in the slow tier.

There are some out there that would simply say "just demote more
aggressively", so this is useful information for the discussion.

+/- ~2% despite greater meomry migration is an excellent result

> > Can you also provide the DRAM-only results for each test?  Presumably,
> > as workload size increases from 440G to 500G, the system probably starts
> > using some amount of swap/zswap/whatever.  It would be good to know how
> > this system compares to swap small amounts of overflow.
> 
> It looks like my explanation doesn't correctly inform you.   The size
> from 440GB to 500GB is for pre allocated cold data to give memory
> pressure on the system so that redis-server cannot be fully allocated at
> fast DRAM, then partially allocated at CXL memory as well.
> 

Yes, sorry for the misunderstanding.  This makes it much clearer.

> 
> I hope my explanation is helpful for you to understand.  Please let me
> know if you have more questions.
>

Excellent work, exciting results! Thank you for the additional answers
:]

~Gregory