Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <0a92d0040edb3b74ac259062d241b8cd28924edf.camel@intel.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces
From:   "ying.huang@intel.com" <ying.huang@intel.com>
To:     Wei Xu <weixugc@google.com>
Cc:     "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        Alistair Popple <apopple@nvidia.com>,
        Yang Shi <shy828301@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Linux MM <linux-mm@kvack.org>,
        Greg Thelen <gthelen@google.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Michal Hocko <mhocko@kernel.org>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Feng Tang <feng.tang@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Tim Chen <tim.c.chen@linux.intel.com>
Date:   Thu, 12 May 2022 09:42:24 +0800
In-Reply-To: <CAAPL-u80BFYTKK=0HRBXOeDTULyPOtbgu5V3fEYDOczTMxgJ1g@mail.gmail.com>
References: <CAAPL-u9sVx94ACSuCVN8V0tKp+AMxiY89cro0japtyB=xNfNBw@mail.gmail.com>
         <CAHbLzkq1YXXLMiREpGnzhJjPssu4WpSsnkTmrLJ=hAEhZVUr9w@mail.gmail.com>
         <CAAPL-u-r2Pc_MaHQmKKNH_icAa_fH1COWb5qSPpr8xffREQ_cQ@mail.gmail.com>
         <87tua3h5r1.fsf@nvdebian.thelocal>
         <CAAPL-u-0HwL6p1SA73LPfFyywG55QqE9O+q=83fhShoJAVVxyQ@mail.gmail.com>
         <875ymerl81.fsf@nvdebian.thelocal> <87fslhhb2l.fsf@linux.ibm.com>
         <CAAPL-u9FvCfgA7xsqStLNZ=W03iyWBmvHrpVzPKyitsGN2v_KQ@mail.gmail.com>
         <68333b21a58604f3fd0e660f1a39921ae22849d8.camel@intel.com>
         <CAAPL-u80BFYTKK=0HRBXOeDTULyPOtbgu5V3fEYDOczTMxgJ1g@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.3-1 
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Wed, 2022-05-11 at 10:07 -0700, Wei Xu wrote:
> On Wed, May 11, 2022 at 12:49 AM ying.huang@intel.com
> <ying.huang@intel.com> wrote:
> > 
> > On Tue, 2022-05-10 at 22:30 -0700, Wei Xu wrote:
> > > On Tue, May 10, 2022 at 4:38 AM Aneesh Kumar K.V
> > > <aneesh.kumar@linux.ibm.com> wrote:
> > > > 
> > > > Alistair Popple <apopple@nvidia.com> writes:
> > > > 
> > > > > Wei Xu <weixugc@google.com> writes:
> > > > > 
> > > > > > On Thu, May 5, 2022 at 5:19 PM Alistair Popple <apopple@nvidia.com> wrote:
> > > > > > > 
> > > > > > > Wei Xu <weixugc@google.com> writes:
> > > > > > > 
> > > > > > > [...]
> > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > Tiering Hierarchy Initialization
> > > > > > > > > > `=============================='
> > > > > > > > > > 
> > > > > > > > > > By default, all memory nodes are in the top tier (N_TOPTIER_MEMORY).
> > > > > > > > > > 
> > > > > > > > > > A device driver can remove its memory nodes from the top tier, e.g.
> > > > > > > > > > a dax driver can remove PMEM nodes from the top tier.
> > > > > > > > > 
> > > > > > > > > With the topology built by firmware we should not need this.
> > > > > > > 
> > > > > > > I agree that in an ideal world the hierarchy should be built by firmware based
> > > > > > > on something like the HMAT. But I also think being able to override this will be
> > > > > > > useful in getting there. Therefore a way of overriding the generated hierarchy
> > > > > > > would be good, either via sysfs or kernel boot parameter if we don't want to
> > > > > > > commit to a particular user interface now.
> > > > > > > 
> > > > > > > However I'm less sure letting device-drivers override this is a good idea. How
> > > > > > > for example would a GPU driver make sure it's node is in the top tier? By moving
> > > > > > > every node that the driver does not know about out of N_TOPTIER_MEMORY? That
> > > > > > > could get messy if say there were two drivers both of which wanted their node to
> > > > > > > be in the top tier.
> > > > > > 
> > > > > > The suggestion is to allow a device driver to opt out its memory
> > > > > > devices from the top-tier, not the other way around.
> > > > > 
> > > > > So how would demotion work in the case of accelerators then? In that
> > > > > case we would want GPU memory to demote to DRAM, but that won't happen
> > > > > if both DRAM and GPU memory are in N_TOPTIER_MEMORY and it seems the
> > > > > only override available with this proposal would move GPU memory into a
> > > > > lower tier, which is the opposite of what's needed there.
> > > > 
> > > > How about we do 3 tiers now. dax kmem devices can be registered to
> > > > tier 3. By default all numa nodes can be registered at tier 2 and HBM or
> > > > GPU can be enabled to register at tier 1. ?
> > > 
> > > This makes sense.  I will send an updated RFC based on the discussions so far.
> > 
> > Are these tier number fixed?  If so, it appears strange that the
> > smallest tier number is 0 on some machines, but 1 on some other
> > machines.
> 
> When the kernel is configured to allow 3 tiers, we can always show all
> the 3 tiers. It is just that some tiers (e.g. tier 0) may be empty on
> some machines.

I still think that it's better to have no empty tiers for auto-generated
memory tiers by kernel.  Yes, the tier number will be not absolutely
stable, but that only happens during system bootup in practice, so it's
not a big issue IMHO.

And, I still think it's better to make only N-1 tiers writable for
totally N tiers (or even readable).  Considering "tier0" is written, how
to deal with nodes in "tier0" before but not after writing?  One
possible way is to put them into "tierN".  And during a user customize
the tiers, the union of "N tiers" may be not complete.

> BTW, the userspace should not assume a specific meaning of a
> particular tier id because it can change depending on the number of
> tiers that the kernel is configured with.  For example, the userspace
> should not assume that tier-2 always means PMEM nodes.  In a system
> with 4 tiers, PMEM nodes may be in tier-3, not tier-2.

Yes.  This sounds good.

Best Regards,
Huang, Ying