Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <CAAPL-u-DGLcKRVDnChN9ZhxPkfxQvz9Sb93kVoX_4J2oiJSkUw@mail.gmail.com>
 <56b41ce6922ed5f640d9bd46a603fa27576532a9.camel@intel.com>
 <c5d55131-76ff-7354-2954-7cfac365a9a5@linux.ibm.com> <CAAPL-u-g86QqHaHGGtVJMER8ENC2dpekK+2qOkxoRFmC0F_80g@mail.gmail.com>
 <87y1z7jj85.fsf@linux.ibm.com> <CAAPL-u90Gzrepbsdpc62FGRBYOx1LAVwZWHg1E7UyVD_CmcROw@mail.gmail.com>
 <333438929a8680317bdae02da921d1e50fb9b0b3.camel@linux.intel.com>
In-Reply-To: <333438929a8680317bdae02da921d1e50fb9b0b3.camel@linux.intel.com>
From:   Wei Xu <weixugc@google.com>
Date:   Thu, 12 May 2022 14:31:49 -0700
Message-ID: <CAAPL-u-nhj-JBH8xA1+BjuC3A_kfd5FSv2qKCL3g71h-cu0WLg@mail.gmail.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2)
To:     Tim Chen <tim.c.chen@linux.intel.com>
Cc:     "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        "ying.huang@intel.com" <ying.huang@intel.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Greg Thelen <gthelen@google.com>,
        Yang Shi <shy828301@gmail.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Michal Hocko <mhocko@kernel.org>,
        Tim C Chen <tim.c.chen@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Alistair Popple <apopple@nvidia.com>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Feng Tang <feng.tang@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Dan Williams <dan.j.williams@intel.com>,
        David Rientjes <rientjes@google.com>,
        Linux MM <linux-mm@kvack.org>,
        Brice Goglin <brice.goglin@gmail.com>,
        Hesham Almatary <hesham.almatary@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, May 12, 2022 at 2:13 PM Tim Chen <tim.c.chen@linux.intel.com> wrote:
>
> On Thu, 2022-05-12 at 01:15 -0700, Wei Xu wrote:
> >
> > I am OK with moving back the memory tier nodelist into node/.  When
> > there are more memory tier attributes needed, we can then create the
> > memory tier subtree and replace the tier nodelist in node/ with
> > symlinks.
> >
> > So the revised sysfs interfaces are:
> >
> > * /sys/devices/system/node/memory_tierN (read-only)
> >
> >   where N = 0, 1, 2
> >
> >   Format: node_list
> >
> > * /sys/devices/system/node/nodeN/memory_tier (read/write)
> >
> >   where N = 0, 1, ...
> >
> >   Format: int or empty
>
> This looks good to me.  Just wonder if having just 1 tier
> lower than DRAM is sufficient. We could have wide performance
> range for such secondary memories and is one tier sufficient for them?
>
> Tim

The tier design can be extended to more than 3 tiers (e.g. via
CONFIG_MAX_MEMORY_TIERS).  MAX_MEMORY_TIERS is set to 3 for now
because without enough memory device performance information provided
by the firmware, it is difficult for the kernel to properly initialize
the memory tier hierarchy beyond 3 tiers (GPU, DRAM, PMEM).  We will
have to resort to the userspace override to set up such many-tier
systems.