Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Huang, Ying" <ying.huang@intel.com>
To:     "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc:     Johannes Weiner <hannes@cmpxchg.org>, linux-mm@kvack.org,
        akpm@linux-foundation.org, Wei Xu <weixugc@google.com>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>, jvgediya.oss@gmail.com
Subject: Re: [PATCH v8 00/12] mm/demotion: Memory tiers and demotion
References: <20220704070612.299585-1-aneesh.kumar@linux.ibm.com>
        <87r130b2rh.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <60e97fa2-0b89-cf42-5307-5a57c956f741@linux.ibm.com>
        <87r12r5dwu.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <0a55e48a-b4b7-4477-a72f-73644b5fc4cb@linux.ibm.com>
        <87mtde6cla.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <fef35622-0bd4-f220-26bd-37d8e0112c4d@linux.ibm.com>
        <87ilo267jl.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <ad4b095b-bb85-b01f-5d69-383219384c29@linux.ibm.com>
        <87edyp67m1.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <878roxuz9l.fsf@linux.ibm.com>
Date:   Thu, 14 Jul 2022 12:56:55 +0800
In-Reply-To: <878roxuz9l.fsf@linux.ibm.com> (Aneesh Kumar K. V.'s message of
        "Wed, 13 Jul 2022 15:10:06 +0530")
Message-ID: <87o7xs47hk.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii
Precedence: bulk

"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:

> "Huang, Ying" <ying.huang@intel.com> writes:

[snip]

>>
>> I believe that sparse memory tier IDs can make memory tier more stable
>> in some cases.  But this is different from the system suggested by
>> Johannes.  Per my understanding, with Johannes' system, we will
>>
>> - one driver may online different memory types (such as kmem_dax may
>>   online HBM, PMEM, etc.)
>>
>> - one memory type manages several memory nodes (NUMA nodes)
>>
>> - one "abstract distance" for each memory type
>>
>> - the "abstract distance" can be offset by user space override knob
>>
>> - memory tiers generated dynamic from different memory types according
>>   "abstract distance" and overridden "offset"
>>
>> - the granularity to group several memory types into one memory tier can
>>   be overridden via user space knob
>>
>> In this way, the memory tiers may be changed totally after user space
>> overridden.  It may be hard to link memory tiers before/after the
>> overridden.  So we may need to reset all per-memory-tier configuration,
>> such as cgroup paritation limit or interleave weight, etc.
>
> Making sure we all agree on the details.
>
> In the proposal https://lore.kernel.org/linux-mm/7b72ccf4-f4ae-cb4e-f411-74d055482026@linux.ibm.com
> instead of calling it "abstract distance" I was referring it as device
> attributes.
>
> Johannes also suggested these device attributes/"abstract distance"
> to be used to derive the memory tier to which the memory type/memory
> device will be assigned.
>
> So dax kmem would manage different types of memory and based on the device
> attributes, we would assign them to different memory tiers (memory tiers
> in the range [0-200)).
>
> Now the additional detail here is that we might add knobs that will be
> used by dax kmem to fine-tune memory types to memory tiers assignment.
> On updating these knob values, the kernel should rebuild the entire
> memory tier hierarchy. (earlier I was considering only newly added
> memory devices will get impacted by such a change. But I agree it
> makes sense to rebuild the entire hierarchy again) But that rebuilding
> will be restricted to dax kmem driver.
>

Thanks for explanation and pointer.  Per my understanding, memory
types and memory devices including abstract distances are used to
describe the *physical* memory devices, not *policy*.  We may add more
physical attributes to these memory devices, such as, latency,
throughput, etc.  I think we can reach consensus on this point?

In contrast, memory tiers are more about policy, such as
demotion/promotion, interleaving and possible partition among cgroups.
How to derive memory tiers from memory types (or devices)?  We have
multiple choices.

Per my understanding, Johannes suggested to use some policy parameters
such as distance granularity (e.g., if granularity is 100, then memory
devices with abstract distance 0-100, 100-200, 200-300, ... will be put
to memory tier 0, 1, 2, ...) to build the memory tiers.  Distance
granularity may be not flexible enough, we may need something like a set
of cutoffs or range, e.g., 50, 100, 200, 500, or 0-50, 50-100, 100-200,
200-500, >500.  These policy parameters should be overridable from user
space.

And per my understanding, you suggested to place memory devices to
memory tiers directly via a knob of memory types (or memory devices).
e.g., memory_type/memtier can be written to place the memory devices of
the memory_type to the specified memtier.  Or via
memorty_type/distance_offset to do that.

Best Regards,
Huang, Ying

[snip]