Received-SPF: pass (google.com: domain of linux-kernel+bounces-21738-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
From: "Huang, Ying" <ying.huang@intel.com>
To: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
Cc: Gregory Price <gregory.price@memverge.com>,  Srinivasulu Thanneeru
 <sthanneeru@micron.com>,  Srinivasulu Opensrc
 <sthanneeru.opensrc@micron.com>,  "linux-cxl@vger.kernel.org"
 <linux-cxl@vger.kernel.org>,  "linux-mm@kvack.org" <linux-mm@kvack.org>,
  "aneesh.kumar@linux.ibm.com" <aneesh.kumar@linux.ibm.com>,
  "dan.j.williams@intel.com" <dan.j.williams@intel.com>,  "mhocko@suse.com"
 <mhocko@suse.com>,  "tj@kernel.org" <tj@kernel.org>,
  "john@jagalactic.com" <john@jagalactic.com>,  Eishan Mirakhur
 <emirakhur@micron.com>,  Vinicius Tavares Petrucci
 <vtavarespetr@micron.com>,  Ravis OpenSrc <Ravis.OpenSrc@micron.com>,
  "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,  "Johannes
 Weiner" <hannes@cmpxchg.org>,  Wei Xu <weixugc@google.com>,  Hao Xiang
 <hao.xiang@bytedance.com>,  "Ho-Ren (Jack) Chuang"
 <horenchuang@bytedance.com>
Subject: Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory tiers
In-Reply-To: <20240109155049.00003f13@Huawei.com> (Jonathan Cameron's message
	of "Tue, 9 Jan 2024 15:50:49 +0000")
References: <20231213175329.594-1-sthanneeru.opensrc@micron.com>
	<87cyv8qcqk.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<ZXyQIJOim1+tE0Qr@memverge.com>
	<87fs00njft.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<PH0PR08MB7955E9F08CCB64F23963B5C3A860A@PH0PR08MB7955.namprd08.prod.outlook.com>
	<87edezc5l1.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<PH0PR08MB79550922630FEC47E4B4D3A3A860A@PH0PR08MB7955.namprd08.prod.outlook.com>
	<87a5pmddl5.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<PH0PR08MB79552F35351FA57EF4BD64B4A860A@PH0PR08MB7955.namprd08.prod.outlook.com>
	<87wmspbpma.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<ZZwrIoP9+ey7rp3C@memverge.com>
	<87o7dv897s.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<20240109155049.00003f13@Huawei.com>
Date: Wed, 10 Jan 2024 14:06:44 +0800
Message-ID: <874jfl90y3.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13)
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii

Jonathan Cameron <Jonathan.Cameron@Huawei.com> writes:

> On Tue, 09 Jan 2024 11:41:11 +0800
> "Huang, Ying" <ying.huang@intel.com> wrote:
>
>> Gregory Price <gregory.price@memverge.com> writes:
>> 
>> > On Thu, Jan 04, 2024 at 02:05:01PM +0800, Huang, Ying wrote:  
>> >> >
>> >> > From  https://lpc.events/event/16/contributions/1209/attachments/1042/1995/Live%20In%20a%20World%20With%20Multiple%20Memory%20Types.pdf
>> >> > abstract_distance_offset: override by users to deal with firmware issue.
>> >> >
>> >> > say firmware can configure the cxl node into wrong tiers, similar to
>> >> > that it may also configure all cxl nodes into single memtype, hence
>> >> > all these nodes can fall into a single wrong tier.
>> >> > In this case, per node adistance_offset would be good to have ?  
>> >> 
>> >> I think that it's better to fix the error firmware if possible.  And
>> >> these are only theoretical, not practical issues.  Do you have some
>> >> practical issues?
>> >> 
>> >> I understand that users may want to move nodes between memory tiers for
>> >> different policy choices.  For that, memory_type based adistance_offset
>> >> should be good.
>> >>   
>> >
>> > There's actually an affirmative case to change memory tiering to allow
>> > either movement of nodes between tiers, or at least base placement on
>> > HMAT information. Preferably, membership would be changable to allow
>> > hotplug/DCD to be managed (there's no guarantee that the memory passed
>> > through will always be what HMAT says on initial boot).  
>> 
>> IIUC, from Jonathan Cameron as below, the performance of memory
>> shouldn't change even for DCD devices.
>> 
>> https://lore.kernel.org/linux-mm/20231103141636.000007e4@Huawei.com/
>> 
>> It's possible to change the performance of a NUMA node changed, if we
>> hot-remove a memory device, then hot-add another different memory
>> device.  It's hoped that the CDAT changes too.
>
> Not supported, but ACPI has _HMA methods to in theory allow changing
> HMAT values based on firmware notifications...  So we 'could' make
> it work for HMAT based description.
>
> Ultimately my current thinking is we'll end up emulating CXL type3
> devices (hiding topology complexity) and you can update CDAT but
> IIRC that is only meant to be for degraded situations - so if you
> want multiple performance regions, CDAT should describe them form the start.

Thank you very much for input!  So, to support degraded performance, we
will need to move a NUMA node between memory tiers.  And, per my
understanding, we should do that in kernel.

>> 
>> So, all in all, HMAT + CDAT can help us to put the memory device in
>> appropriate memory tiers.  Now, we have HMAT support in upstream.  We
>> will working on CDAT support.
>> 

--
Best Regards,
Huang, Ying