Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <68dd0ea6-3a02-2bcc-cb63-ddd633428685@linux.ibm.com>
Date:   Fri, 27 May 2022 14:56:53 +0530
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2)
Content-Language: en-US
To:     Wei Xu <weixugc@google.com>
Cc:     Ying Huang <ying.huang@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Greg Thelen <gthelen@google.com>,
        Yang Shi <shy828301@gmail.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Michal Hocko <mhocko@kernel.org>,
        Tim C Chen <tim.c.chen@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Alistair Popple <apopple@nvidia.com>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Feng Tang <feng.tang@intel.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Dan Williams <dan.j.williams@intel.com>,
        David Rientjes <rientjes@google.com>,
        Linux MM <linux-mm@kvack.org>,
        Brice Goglin <brice.goglin@gmail.com>,
        Hesham Almatary <hesham.almatary@huawei.com>
References: <CAAPL-u-DGLcKRVDnChN9ZhxPkfxQvz9Sb93kVoX_4J2oiJSkUw@mail.gmail.com>
 <20220512160010.00005bc4@Huawei.com>
 <CAAPL-u_diGYEb7+WsgqNBLRix-nRCk2SsDj6p9r8j5JZwOABZQ@mail.gmail.com>
 <6b7c472b50049592cde912f04ca47c696caa2227.camel@intel.com>
 <CAAPL-u-NAJkSXHzQr8OtEEGnnUrsCE9US6c5S5Rv0xutASxv1Q@mail.gmail.com>
 <ad1bc96a86b21b8f46d356f0c0b7449be38b2220.camel@intel.com>
 <CAAPL-u8XzJnrew4vuFq7GgwgjjBff8MwRFGTDmUc9DOwa+q=FA@mail.gmail.com>
 <6ce724e5c67d4f7530457897fa08d0a8ba5dd6d0.camel@intel.com>
 <594046f8-9ab3-786a-fc48-8a61f1238f52@linux.ibm.com>
 <CAAPL-u-3-iOoSK07TE=gUN8a+TsXNNEiFaEArf7+zvTgxDJnRQ@mail.gmail.com>
From:   Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>
In-Reply-To: <CAAPL-u-3-iOoSK07TE=gUN8a+TsXNNEiFaEArf7+zvTgxDJnRQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 5/25/22 10:57 PM, Wei Xu wrote:
> On Wed, May 25, 2022 at 3:01 AM Aneesh Kumar K V
> <aneesh.kumar@linux.ibm.com> wrote:
>>
>> On 5/25/22 2:33 PM, Ying Huang wrote:
>>> On Tue, 2022-05-24 at 22:32 -0700, Wei Xu wrote:
>>>> On Tue, May 24, 2022 at 1:24 AM Ying Huang <ying.huang@intel.com> wrote:
>>>>>
>>>>> On Tue, 2022-05-24 at 00:04 -0700, Wei Xu wrote:
>>>>>> On Thu, May 19, 2022 at 8:06 PM Ying Huang <ying.huang@intel.com> wrote:
>>>>>>>
>>
>> ...
>>
>>>
>>> OK.  Just to confirm.  Does this mean that we will have fixed device ID,
>>> for example,
>>>
>>> GPU                   memtier255
>>> DRAM (with CPU)               memtier0
>>> PMEM                  memtier1
>>>
>>> When we add a new memtier, it can be memtier254, or memter2?  The rank
>>> value will determine the real demotion order.
>>>
>>> I think you may need to send v3 to make sure everyone is at the same
>>> page.
>>>
>>
>> What we have implemented which we will send as RFC shortly is below.
>>
>> cd /sys/dekvaneesh@ubuntu-guest:~$ cd /sys/devices/system/
>> kvaneesh@ubuntu-guest:/sys/devices/system$ pwd
>> /sys/devices/system
>> kvaneesh@ubuntu-guest:/sys/devices/system$ ls
>> clockevents  clocksource  container  cpu  edac  memory  memtier  mpic
>> node  power
>> kvaneesh@ubuntu-guest:/sys/devices/system$ cd memtier/
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ pwd
>> /sys/devices/system/memtier
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ ls
>> default_rank  max_rank  memtier1  power  uevent
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cat default_rank
>> 1
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cat max_rank
>> 3
> 
> For flexibility, we don't want max_rank to be interpreted as the
> number of memory tiers.  Also, we want to leave spaces in rank values
> to allow new memtiers to be inserted when needed.  So I'd suggest to
> make max_rank a much larger value (e.g. 255).
> 
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cd memtier1/
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ ls
>> nodelist  power  rank  subsystem  uevent
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cat nodelist
>> 0-3
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cat rank
>> 1
>> kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cd
>> ../../node/node1/
>> kvaneesh@ubuntu-guest:/sys/devices/system/node/node1$ cat memtier
>> 1
>> kvaneesh@ubuntu-guest:/sys/devices/system/node/node1$
>> root@ubuntu-guest:/sys/devices/system/node/node1# echo 0 > memtier
>> root@ubuntu-guest:/sys/devices/system/node/node1# cat memtier
>> 0
>> root@ubuntu-guest:/sys/devices/system/node/node1# cd ../../memtier/
>> root@ubuntu-guest:/sys/devices/system/memtier# ls
>> default_rank  max_rank  memtier0  memtier1  power  uevent
>> root@ubuntu-guest:/sys/devices/system/memtier# cd memtier0/
>> root@ubuntu-guest:/sys/devices/system/memtier/memtier0# cat nodelist
>> 1
>> root@ubuntu-guest:/sys/devices/system/memtier/memtier0# cat rank
>> 0
> 
> It looks like the example here demonstrates the dynamic creation of
> memtier0.  If so, how is the rank of memtier0 determined?  If we want
> to support creating new memtiers at runtime, I think an explicit
> interface that specifies both device ID and rank is preferred to avoid
> implicit dependencies between device IDs and ranks.
> 

Right now to keep it all simpler there is a 1:1 relation ship between 
memory tier and rank value. ie.

memory tier  rank
memtier0     100
memtier1     200
memtier2     300

Currently we are limiting this to max 3 tiers. Hence the above is very 
easy. Once we really get dynamic tier creation, we should be looking at 
creating a new memory tier with highest possible rank value. Once we 
establish the memory tier, we then modify the rank value to a desired 
value. There will be a kernel interface to add a node to a memory tier 
with specific rank value so drivers can do that if required.

I haven't gone to that implementation because i was hoping we could get 
to that later when we really start requiring dynamic tier support.

I will share the patch series we have been working with. I am yet to get 
the documentation added. But then i will not wait for it to be complete 
so that we can get some early testing/feedback.

-aneesh