Received-SPF: pass (google.com: domain of linux-kernel+bounces-1438-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Fri, 15 Dec 2023 12:42:56 -0500
From: Gregory Price <gregory.price@memverge.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: sthanneeru.opensrc@micron.com, linux-cxl@vger.kernel.org,
	linux-mm@kvack.org, sthanneeru@micron.com,
	aneesh.kumar@linux.ibm.com, dan.j.williams@intel.com,
	mhocko@suse.com, tj@kernel.org, john@jagalactic.com,
	emirakhur@micron.com, vtavarespetr@micron.com,
	Ravis.OpenSrc@micron.com, Jonathan.Cameron@huawei.com,
	linux-kernel@vger.kernel.org
Subject: Re: [RFC PATCH v2 0/2] Node migration between memory tiers
Message-ID: <ZXyQIJOim1+tE0Qr@memverge.com>
References: <20231213175329.594-1-sthanneeru.opensrc@micron.com>
 <87cyv8qcqk.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87cyv8qcqk.fsf@yhuang6-desk2.ccr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?LvHmO1X/jLL9+s8cBAU9ANDDX2acNVj9CPGN+MNUM7u+Ryd6O5R19AaRpaXZ?=
 =?us-ascii?Q?yKhm/kQO9zaDu2IpOIybDh0nYoaod56qOU/7ei+C2Pul0XZ81nQM70d9Upa5?=
 =?us-ascii?Q?zXFLw6jXuk0+vbv+YfmGlWMCgUKxRgv1i6082qEfvI/v3XD/1jk9KK0oaaFO?=
 =?us-ascii?Q?B4M/VuAB+XVp/v6v6R0KdbF+laYGf3Npad7jSfyvYoyhLBYCa5jxMGtdTVXe?=
 =?us-ascii?Q?0k4LzU0yrIvWRr6glDNKuNIwTSCnXAbf+OvbSfu1ToV+4IlshrciJxWXIRoS?=
 =?us-ascii?Q?yEZQgjFUvV/E80jKSgnqrpXzSiS3jaqoiD6gz7an4/4eJeOkde1qbxNEpDP7?=
 =?us-ascii?Q?oWEehBy/cKHDiV6Jm7boL9DMSQkHFgtYYSakq13J4JoIBmv924+Nf0sHJ4nA?=
 =?us-ascii?Q?lr4zTVyOjRi4/uP31p3naYMxDYzLI/3ckapCcnLe7FF9kVP+KLoPmTUJwhhO?=
 =?us-ascii?Q?Gn7yupmz+XqpeaaoO/PmGyHAyIq1FalYnmbbB5Bg2Bp0/kDzA1+ajYS8lrZL?=
 =?us-ascii?Q?B+PZvP0Bb3BsqSHipIc1S+Ta9UPpTmnPNxx+OX79LuLgf7yMPTW2MqiHd0u7?=
 =?us-ascii?Q?83GaiuTvHgJ11iNH45HjxFDHmNvIWibVIG97dy04jQtZDnwF/r4ZTTl5W7EC?=
 =?us-ascii?Q?ihL4BO2bdwPuZQpBgMln47Oe+2ewQhjAosRzetTbgK2+F74MOkCGhIKENss0?=
 =?us-ascii?Q?P+WT7p5cuQYve3XdXopXmm/dnWS46P/v6J3rhBU/DHe+DsMQ3mCfTZRn36OS?=
 =?us-ascii?Q?3tEiyr/EpLbgktGQ3cwWaTz+l7JRzF/JA28f5rVFktGHIOKpbkYGvzDq+fyD?=
 =?us-ascii?Q?hM4QtJr21vm/JNHsE2UIOTyui4Zs0dTmDjjV/99RkE07rtsLmdPOzIa74D3R?=
 =?us-ascii?Q?Ky5yxeXRb9tIlKJl5xrW5QtiSkWuYDQcfjwTFGWjLCcVuxbyQNXXE7BrPl1g?=
 =?us-ascii?Q?YU3LC3bnEcobkuhJ+k9JHaRZLh+PcdVwGZltoMfOhWIiz2OxNTIAxMaEzwkn?=
 =?us-ascii?Q?3iNL+FWgr6HAGFON+j8r2FcCLmEsekTrRExTDN4v8r2AmEuSbFZY5yfkVsbd?=
 =?us-ascii?Q?BITaZZkzHiHTEGUb8scYzJzAv7al7IoDGBJ7MbPgbPohH8/tZznOvgqEkl8x?=
 =?us-ascii?Q?S8JQMZA3DYrSG+xzpUb+KqnuChe+TAK3G0NqAIHky3VVhHPnkKW5E2RDMx8D?=
 =?us-ascii?Q?mJH5zVtgWmuEyvw9Eg72vJSdt47LGB3841UvHFBo8re1VR2/sZh8j92PIjEI?=
 =?us-ascii?Q?YFzdkeQi+521K2GidDs0ucrm5DaUOSvxAIA4qVhrNBzI23wYhapmImjRAowZ?=
 =?us-ascii?Q?1alSfHyeH/RJUhMI8Royn12gztw6+9hv5X1SCgnpy+r4MLdZib+U0roliJxg?=
 =?us-ascii?Q?oikYBK6zwZjZn7Wdnk2TPPN22AB7D68w0X/6LTe1qRWq1gmloIZ9TJbtdmJy?=
 =?us-ascii?Q?9LUUNr2z3LDNynTMZ7CWAJnOBZ62wIU/l1Td/woX6LzIA5BL8NSkY6riQ8hE?=
 =?us-ascii?Q?ETg2z6WJH/EVqE6fS3od0FQu23aoRSPOPr7NdbW+oCrHQzme0V5Vtd7V6U2O?=
 =?us-ascii?Q?DQEynX1dE9rsP5bbImodYBGFuDroGjDrFmmpaoDq2vWiD0TuL6Zy/fE3EoBR?=
 =?us-ascii?Q?WA=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c17b7672-beb9-4748-0847-08dbfd954a49
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Dec 2023 17:43:04.3528
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +tyNjT48BM55/16fj+TVk5bU+ZTcsTnQeJZsBgCABQB17Rsa5xcTjoprr1B2x/ouiZlDAwGx4CGG+PDGEso2o+vz3+1TpZIh2FudEdK8Zb0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR17MB3873

On Fri, Dec 15, 2023 at 01:02:59PM +0800, Huang, Ying wrote:
> <sthanneeru.opensrc@micron.com> writes:
> 
> > =============
> > Version Notes:
> >
> > V2 : Changed interface to memtier_override from adistance_offset.
> > memtier_override was recommended by
> > 1. John Groves <john@jagalactic.com>
> > 2. Ravi Shankar <ravis.opensrc@micron.com>
> > 3. Brice Goglin <Brice.Goglin@inria.fr>
> 
> It appears that you ignored my comments for V1 as follows ...
> 
> https://lore.kernel.org/lkml/87o7f62vur.fsf@yhuang6-desk2.ccr.corp.intel.com/
> https://lore.kernel.org/lkml/87jzpt2ft5.fsf@yhuang6-desk2.ccr.corp.intel.com/
> https://lore.kernel.org/lkml/87a5qp2et0.fsf@yhuang6-desk2.ccr.corp.intel.com/
> 

Not speaking for the group, just chiming in because i'd discussed it
with them.

"Memory Type" is a bit nebulous.  Is a Micron Type-3 with performance X
and an SK Hynix Type-3 with performance Y a "Different type", or are
they the "Same Type" given that they're both Type 3 backed by some form
of DDR?  Is socket placement of those devices relevant for determining
"Type"?  Is whether they are behind a switch relevant for determining
"Type"? "Type" is frustrating when everything we're talking about
managing is "Type-3" with difference performance.

A concrete example:
To the system, a Multi-Headed Single Logical Device (MH-SLD) looks
exactly the same as an standard SLD.  I may want to have some
combination of local memory expansion devices on the majority of my
expansion slots, but reserve 1 slot on each socket for a connection to
the MH-SLD.   As of right now: There is no good way to differentiate the
devices in terms of "Type" - and even if you had that, the tiering
system would still lump them together.

Similarly, an initial run of switches may or may not allow enumeration
of devices behind it (depends on the configuration), so you may end up
with a static numa node that "looks like" another SLD - despite it being
some definition of "GFAM".  Do number of hops matter in determining
"Type"?

So I really don't think "Type" is useful for determining tier placement.

As of right now, the system lumps DRAM nodes as one tier, and pretty
much everything else as "the other tier". To me, this patch set is an
initial pass meant to allow user-control over tier composition while
the internal mechanism is sussed out and the environment develops.

In general, a release valve that lets you redefine tiers is very welcome
for testing and validation of different setups while the industry evolves.

Just my two cents.

~Gregory

> --
> Best Regards,
> Huang, Ying
>