Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220812055710.357820-1-aneesh.kumar@linux.ibm.com>
 <20220812055710.357820-5-aneesh.kumar@linux.ibm.com> <87wnbacjsh.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <cd1c13ee-6fc3-bde8-96f9-8c3c93441275@linux.ibm.com> <CAC=cRTMZZ9bqyC7pnxD1zUWqfBiQ9U7im+8EYa_8GVK8iA7HXQ@mail.gmail.com>
 <e5fbaf30-1f97-63de-a9a5-2ae5359120ff@amd.com>
In-Reply-To: <e5fbaf30-1f97-63de-a9a5-2ae5359120ff@amd.com>
From:   huang ying <huang.ying.caritas@gmail.com>
Date:   Tue, 16 Aug 2022 16:26:02 +0800
Message-ID: <CAC=cRTO=+zdKGHRMLpzg2PfJ2rPSQL+xoqA5sAkafaaTYHPr+w@mail.gmail.com>
Subject: Re: [PATCH v14 04/10] mm/demotion/dax/kmem: Set node's abstract
 distance to MEMTIER_DEFAULT_DAX_ADISTANCE
To:     Bharata B Rao <bharata@amd.com>
Cc:     Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>,
        "Huang, Ying" <ying.huang@intel.com>, linux-mm@kvack.org,
        akpm@linux-foundation.org, Wei Xu <weixugc@google.com>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Johannes Weiner <hannes@cmpxchg.org>, jvgediya.oss@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Aug 16, 2022 at 4:12 PM Bharata B Rao <bharata@amd.com> wrote:
>
> On 8/16/2022 12:58 PM, huang ying wrote:
> > On Tue, Aug 16, 2022 at 1:10 PM Aneesh Kumar K V
> > <aneesh.kumar@linux.ibm.com> wrote:
> >>
> >> On 8/15/22 8:09 AM, Huang, Ying wrote:
> >>> "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:
> >>>
> >
> > [snip]
> >
> >>>>
> >>>> +/*
> >>>> + * Default abstract distance assigned to the NUMA node onlined
> >>>> + * by DAX/kmem if the low level platform driver didn't initialize
> >>>> + * one for this NUMA node.
> >>>> + */
> >>>> +#define MEMTIER_DEFAULT_DAX_ADISTANCE       (MEMTIER_ADISTANCE_DRAM=
 * 2)
> >>>
> >>> If my understanding were correct, this is targeting Optane DCPMM for
> >>> now.  The measured results in the following paper is,
> >>>
> >>> https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fa=
rxiv.org%2Fpdf%2F2002.06018.pdf&amp;data=3D05%7C01%7Cbharata%40amd.com%7C1c=
5015b55ff849e5275408da7f58e67d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7=
C637962317187856589%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l=
uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=3DSxSC8WaKEeT=
yfZXoqtI%2FZAoBXXp82PnTeyyavrV%2FGGg%3D&amp;reserved=3D0
> >>>
> >>> Section: 2.1 Read/Write Latencies
> >>>
> >>> "
> >>> For read access, the latency of DCPMM was 400.1% higher than that of
> >>> DRAM. For write access, it was 407.1% higher.
> >>> "
> >>>
> >>> Section: 2.2 Read/Write Bandwidths
> >>>
> >>> "
> >>> For read access, the throughput of DCPMM was 37.1% of DRAM. For write
> >>> access, it was 7.8%
> >>> "
> >>>
> >>> According to the above data, I think the MEMTIER_DEFAULT_DAX_ADISTANC=
E
> >>> can be "5 * MEMTIER_ADISTANCE_DRAM".
> >>>
> >>
> >> If we look at mapping every 100% increase in latency as a memory tier,=
 we essentially
> >> will have 4 memory tier here. Each memory tier is covering a range of =
abstract distance 128.
> >> which makes a total adistance increase from MEMTIER_ADISTANCE_DRAM by =
512. This puts
> >> DEFAULT_DAX_DISTANCE at 1024 or  MEMTIER_ADISTANCE_DRAM * 2
> >
> > If my understanding were correct, you are suggesting to use a kind of
> > logarithmic mapping from latency to abstract distance?  That is,
> >
> >   abstract_distance =3D log2(latency)
> >
> > While I am suggesting to use a kind of linear mapping from latency to
> > abstract distance.  That is,
> >
> >   abstract_distance =3D C * latency
> >
> > I think that linear mapping is easy to understand.
> >
> > Are there some good reasons to use logarithmic mapping?
>
> Also, what is the recommendation for using bandwidth measure which
> may be available from HMAT for CXL memory? How is bandwidth going
> to influence the abstract distance?

This is a good question.

Per my understanding, latency stands for idle latency by default.  But
in practice, the latency under some reasonable memory accessing
throughput is the "real" latency.  So the memory with lower bandwidth
should have a larger abstract distance than the memory with higher
bandwidth even if the idle latency is the same.  But I don't have a
perfect formula to combine idle latency and bandwidth into abstract
distance.  One possibility is to increase abstract distance if the
bandwidth of the memory is much lower than that of DRAM.

Best Regards,
Huang, Ying