Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <8589e329-d06d-3be2-55f8-76d4539ea80f@linux.ibm.com>
Date:   Mon, 5 Sep 2022 10:57:50 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.2.0
Subject: Re: [PATCH v3 updated] mm/demotion: Expose memory tier details via
 sysfs
Content-Language: en-US
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Wei Xu <weixugc@google.com>, Johannes Weiner <hannes@cmpxchg.org>,
        Linux MM <linux-mm@kvack.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        jvgediya.oss@gmail.com, Bharata B Rao <bharata@amd.com>,
        Greg Thelen <gthelen@google.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "Rafael J. Wysocki" <rafael@kernel.org>
References: <20220830081736.119281-1-aneesh.kumar@linux.ibm.com>
 <87pmgezkhp.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <CAAPL-u8MEs04DkHy6kaS788VjdYZZjAYOgzMnioOzDXbc0ZhhQ@mail.gmail.com>
 <d91beb53-e940-e02a-f9ca-3326bf914da7@linux.ibm.com>
 <87fshaz63h.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <698120ce-d4df-3d13-dea9-a8f5c298783c@linux.ibm.com>
 <87bkryz4nh.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <2b4ddc45-74ae-27df-d973-6724f61f4e18@linux.ibm.com>
 <877d2mz3c1.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <45488760-02b5-115b-c16d-5219303f2f33@linux.ibm.com>
 <CAAPL-u_UoAQ9koo892sG-Tx4bi4xDRe9PUtjmFSsn90uU-n31g@mail.gmail.com>
 <871qsuyzr2.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <672e528d-40b7-fc12-9b0c-1591d586c079@linux.ibm.com>
 <87wnamxi30.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <5aaf395d-514a-2717-58c6-3845b97692bd@linux.ibm.com>
 <87sfl6y4d0.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <dbd5b8e4-2b74-97cd-f05c-1b57a2df5c0c@linux.ibm.com>
 <87ilm2xv26.fsf@yhuang6-desk2.ccr.corp.intel.com>
From:   Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>
In-Reply-To: <87ilm2xv26.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Precedence: bulk

On 9/5/22 10:43 AM, Huang, Ying wrote:
> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
> 
>> On 9/5/22 7:22 AM, Huang, Ying wrote:
>>> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
>>>
>>>> On 9/2/22 2:34 PM, Huang, Ying wrote:
>>>>> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
>>>>>
>>>>>> On 9/2/22 1:27 PM, Huang, Ying wrote:
>>>>>>> Wei Xu <weixugc@google.com> writes:
>>>>>>>
>>>>>>>> On Thu, Sep 1, 2022 at 11:44 PM Aneesh Kumar K V
>>>>>>>> <aneesh.kumar@linux.ibm.com> wrote:
>>>>>>>>>
>>>>>>>>> On 9/2/22 12:10 PM, Huang, Ying wrote:
>>>>>>>>>> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
>>>>>>>>>>
>>>>>>>>>>> On 9/2/22 11:42 AM, Huang, Ying wrote:
>>>>>>>>>>>> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 9/2/22 11:10 AM, Huang, Ying wrote:
>>>>>>>>>>>>>> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 9/2/22 10:39 AM, Wei Xu wrote:
>>>>>>>>>>>>>>>> On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <ying.huang@intel.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> writes:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 9/1/22 12:31 PM, Huang, Ying wrote:
>>>>>>>>>>>>>>>>>>> "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
>>>>>>>>>>>>>>>>>>>> related details can be found. All allocated memory tiers will be listed
>>>>>>>>>>>>>>>>>>>> there as /sys/devices/virtual/memory_tiering/memory_tierN/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The nodes which are part of a specific memory tier can be listed via
>>>>>>>>>>>>>>>>>>>> /sys/devices/virtual/memory_tiering/memory_tierN/nodes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think "memory_tier" is a better subsystem/bus name than
>>>>>>>>>>>>>>>>>>> memory_tiering.  Because we have a set of memory_tierN devices inside.
>>>>>>>>>>>>>>>>>>> "memory_tier" sounds more natural.  I know this is subjective, just my
>>>>>>>>>>>>>>>>>>> preference.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
>>>>>>>>>>>>>>> because we would want it to a susbsystem where all memory tiering related details can be found
>>>>>>>>>>>>>>> including memory type in the future. This is as per discussion
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't think that it's a good idea to mix 2 types of devices in one
>>>>>>>>>>>>>> subsystem (bus).  If my understanding were correct, that breaks the
>>>>>>>>>>>>>> driver core convention.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> All these are virtual devices .I am not sure i follow what you mean by 2 types of devices.
>>>>>>>>>>>>> memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows
>>>>>>>>>>>>> details of memory tiers and can possibly contain details of different memory types .
>>>>>>>>>>>>
>>>>>>>>>>>> IMHO, memory_tier and memory_type are 2 kind of devices.  They have
>>>>>>>>>>>> almost totally different attributes (sysfs file).  So, we should create
>>>>>>>>>>>> 2 buses for them.  Each has its own attribute group.  "virtual" itself
>>>>>>>>>>>> isn't a subsystem.
>>>>>>>>>>>
>>>>>>>>>>> Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate
>>>>>>>>>>> them within the same subdirectory? I am still not clear why you are suggesting they need to be in different
>>>>>>>>>>> sysfs hierarchy.  It doesn't break any driver core convention as you mentioned earlier.
>>>>>>>>>>>
>>>>>>>>>>> /sys/devices/virtual/memory_tiering/memory_tierN
>>>>>>>>>>> /sys/devices/virtual/memory_tiering/memory_typeN
>>>>>>>>>>
>>>>>>>>>> I think we should add
>>>>>>>>>>
>>>>>>>>>>  /sys/devices/virtual/memory_tier/memory_tierN
>>>>>>>>>>  /sys/devices/virtual/memory_type/memory_typeN
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am trying to find if there is a technical reason to do the same?
>>>>>>>>>
>>>>>>>>>> I don't think this is complex.  Devices of same bus/subsystem should
>>>>>>>>>> have mostly same attributes.  This is my understanding of driver core
>>>>>>>>>> convention.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was not looking at this from code complexity point. Instead of having multiple directories
>>>>>>>>> with details w.r.t memory tiering, I was looking at consolidating the details
>>>>>>>>> within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices
>>>>>>>>> are consolidated within /sys/devics/virtual/).
>>>>>>>>>
>>>>>>>>> -aneesh
>>>>>>>>
>>>>>>>> Here is an example of /sys/bus/nd/devices (I know it is not under
>>>>>>>> /sys/devices/virtual, but it can still serve as a reference):
>>>>>>>>
>>>>>>>> ls -1 /sys/bus/nd/devices
>>>>>>>>
>>>>>>>> namespace2.0
>>>>>>>> namespace3.0
>>>>>>>> ndbus0
>>>>>>>> nmem0
>>>>>>>> nmem1
>>>>>>>> region0
>>>>>>>> region1
>>>>>>>> region2
>>>>>>>> region3
>>>>>>>>
>>>>>>>> So I think it is not unreasonable if we want to group memory tiering
>>>>>>>> related interfaces within a single top directory.
>>>>>>>
>>>>>>> Thanks for pointing this out.  My original understanding of driver core
>>>>>>> isn't correct.
>>>>>>>
>>>>>>> But I still think it's better to separate instead of mixing memory_tier
>>>>>>> and memory_type.  Per my understanding, memory_type shows information
>>>>>>> (abstract distance, latency, bandwidth, etc.) of memory types (and
>>>>>>> nodes), it can be useful even without memory tiers.  That is, memory
>>>>>>> types describes the physical characteristics, while memory tier reflects
>>>>>>> the policy.
>>>>>>>
>>>>>>
>>>>>> The latency and bandwidth details are already exposed via 
>>>>>>
>>>>>> 	/sys/devices/system/node/nodeY/access0/initiators/
>>>>>>
>>>>>> Documentation/admin-guide/mm/numaperf.rst
>>>>>>
>>>>>> That is the interface that libraries like libmemkind will look at for finding
>>>>>> details w.r.t latency/bandwidth
>>>>>
>>>>> Yes.  Only with that, it's still inconvenient to find out which nodes
>>>>> belong to same memory type (has same performance, same topology, managed
>>>>> by same driver, etc).  So memory types can still provide useful
>>>>> information even without memory tiering.
>>>>>
>>>>
>>>> I am not sure i quiet follow what to conclude from your reply. I used the subsystem name
>>>> "memory_tiering" so that all memory tiering related information can be consolidated there.
>>>> I guess you agreed to the above part that we can consolidated things like that. 
>>>
>>> I just prefer to separate memory_tier and memory_type sysfs directories
>>> personally.  Because memory_type describes the physical memory types and
>>> performance, while memory_tier is more about the policy to group
>>> memory_types.
>>>
>> IMHO we can decide on that based on why we end up adding memory_type details to sysfs. If that
>> is only for memory tier modification from userspace we can look at adding that in the memory tiering
>> sysfs hierarchy. 
>>
>> Also since we have precedence of consolidating things within a sysfs hierarchy as explained in previous emails,
>> I think we should keep "memory_tiering" as sysfs subsystem name? I hope we can get an agreement on that
>> for now?
> 
> I prefer to separate memory_tier and memory_type, so the subsystem name
> should be "memory_tier".  You prefer to consolidate memory_tier and
> memory_type, so the subsystem name should be "memory_tiering".
> 
> The main reason behind my idea is that memory_type isn't tied with
> memory tiering directly.  It describes some hardware property.  Even if
> we don't use memory tiering, we can still use that to classify the
> memory devices in the system.
> 
> Why do you want to consolidate them?  To reduce one directory from
> sysfs?
> 

So that it is much intuitive for user to got to memory_tiering sysfs hierarchy
to change the memory tier levels. As I mentioned earlier the reason for consolidating things
is to accommodate the possibility of supporting changing abstract distance of a memory type
so that we can change the memory tier assignment of that specific memory type. I don't
see any other reason we would want to expose memory type to userspace as of now.


> I want to get opinions from other people too.
> 
> Best Regards,
> Huang, Ying

-aneesh