Received-SPF: pass (google.com: domain of linux-kernel+bounces-91742-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: "Huang, Ying" <ying.huang@intel.com>
To: "Ho-Ren (Jack) Chuang" <horenchuang@bytedance.com>
Cc: Gregory Price <gourry.memverge@gmail.com>,  aneesh.kumar@linux.ibm.com,
  mhocko@suse.com,  tj@kernel.org,  john@jagalactic.com,  Eishan Mirakhur
 <emirakhur@micron.com>,  Vinicius Tavares Petrucci
 <vtavarespetr@micron.com>,  Ravis OpenSrc <Ravis.OpenSrc@micron.com>,
  Alistair Popple <apopple@nvidia.com>,  "Rafael J. Wysocki"
 <rafael@kernel.org>,  Len Brown <lenb@kernel.org>,  Andrew Morton
 <akpm@linux-foundation.org>,  Dave Jiang <dave.jiang@intel.com>,  Dan
 Williams <dan.j.williams@intel.com>,  Jonathan Cameron
 <Jonathan.Cameron@huawei.com>,  linux-acpi@vger.kernel.org,
  linux-kernel@vger.kernel.org,  linux-mm@kvack.org,  "Ho-Ren (Jack)
 Chuang" <horenc@vt.edu>,  "Ho-Ren (Jack) Chuang" <horenchuang@gmail.com>,
  linux-cxl@vger.kernel.org,  qemu-devel@nongnu.org
Subject: Re: [External] Re: [PATCH v1 0/1] Improved Memory Tier Creation for
 CPUless NUMA Nodes
In-Reply-To: <CAKPbEqr-0yPDW7qps24vJgVCtVOGy_Jm4kcc0FKUsL3d9APDsw@mail.gmail.com>
	(Ho-Ren Chuang's message of "Mon, 4 Mar 2024 22:22:59 -0800")
References: <20240301082248.3456086-1-horenchuang@bytedance.com>
	<87frx6btqp.fsf@yhuang6-desk2.ccr.corp.intel.com>
	<CAKPbEqr-0yPDW7qps24vJgVCtVOGy_Jm4kcc0FKUsL3d9APDsw@mail.gmail.com>
Date: Tue, 05 Mar 2024 14:34:36 +0800
Message-ID: <87h6hl9og3.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13)
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

"Ho-Ren (Jack) Chuang" <horenchuang@bytedance.com> writes:

> On Sun, Mar 3, 2024 at 6:47=E2=80=AFPM Huang, Ying <ying.huang@intel.com>=
 wrote:
>>
>> "Ho-Ren (Jack) Chuang" <horenchuang@bytedance.com> writes:
>>
>> > The memory tiering component in the kernel is functionally useless for
>> > CPUless memory/non-DRAM devices like CXL1.1 type3 memory because the n=
odes
>> > are lumped together in the DRAM tier.
>> > https://lore.kernel.org/linux-mm/PH0PR08MB7955E9F08CCB64F23963B5C3A860=
A@PH0PR08MB7955.namprd08.prod.outlook.com/T/
>>
>> I think that it's unfair to call it "useless".  Yes, it doesn't work if
>> the CXL memory device are not enumerate via drivers/dax/kmem.c.  So,
>> please be specific about in which cases it doesn't work instead of too
>> general "useless".
>>
>
> Thank you and I didn't mean anything specific. I simply reused phrases
> we discussed
> earlier in the previous patchset. I will change them to the following in =
v2:
> "At boot time, current memory tiering assigns all detected memory nodes
> to the same DRAM tier. This results in CPUless memory/non-DRAM devices,
> such as CXL1.1 type3 memory, being unable to be assigned to the
> correct memory tier,
> leading to the inability to migrate pages between different types of memo=
ry."
>
> Please see if this looks more specific.

I don't think that the description above is accurate.  In fact, there
are 2 ways to enumerate the memory device,

1. Mark it as reserved memory (E820_TYPE_SOFT_RESERVED, etc.) in E820
   table or something similar.

2. Mark it as normal memory (E820_TYPE_RAM) in E820 table or something
   similar

For 1, the memory device (including CXL memory) is onlined via
drivers/dax/kmem.c, so will be put in proper memory tiers.  For 2, the
memory device is indistinguishable with normal DRAM with current
implementation.  And this is what this patch is working on.

Right?

--
Best Regards,
Huang, Ying

>> > This patchset automatically resolves the issues. It delays the initial=
ization
>> > of memory tiers for CPUless NUMA nodes until they obtain HMAT informat=
ion
>> > at boot time, eliminating the need for user intervention.
>> > If no HMAT specified, it falls back to using `default_dram_type`.
>> >
>> > Example usecase:
>> > We have CXL memory on the host, and we create VMs with a new system me=
mory
>> > device backed by host CXL memory. We inject CXL memory performance att=
ributes
>> > through QEMU, and the guest now sees memory nodes with performance att=
ributes
>> > in HMAT. With this change, we enable the guest kernel to construct
>> > the correct memory tiering for the memory nodes.
>> >
>> > Ho-Ren (Jack) Chuang (1):
>> >   memory tier: acpi/hmat: create CPUless memory tiers after obtaining
>> >     HMAT info
>> >
>> >  drivers/acpi/numa/hmat.c     |  3 ++
>> >  include/linux/memory-tiers.h |  6 +++
>> >  mm/memory-tiers.c            | 76 ++++++++++++++++++++++++++++++++----
>> >  3 files changed, 77 insertions(+), 8 deletions(-)
>>
>> --
>> Best Regards,
>> Huang, Ying