Subject: Re: [PATCH v3 0/3] create sysfs representation of ACPI HMAT
To: Ross Zwisler <ross.zwisler@linux.intel.com>,
        linux-kernel@vger.kernel.org
References: <20171214021019.13579-1-ross.zwisler@linux.intel.com>
Cc: "Anaczkowski, Lukasz" <lukasz.anaczkowski@intel.com>,
        "Box, David E" <david.e.box@intel.com>,
        "Kogut, Jaroslaw" <Jaroslaw.Kogut@intel.com>,
        "Koss, Marcin" <marcin.koss@intel.com>,
        "Koziej, Artur" <artur.koziej@intel.com>,
        "Lahtinen, Joonas" <joonas.lahtinen@intel.com>,
        "Moore, Robert" <robert.moore@intel.com>,
        "Nachimuthu, Murugasamy" <murugasamy.nachimuthu@intel.com>,
        "Odzioba, Lukasz" <lukasz.odzioba@intel.com>,
        "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
        "Rafael J. Wysocki" <rjw@rjwysocki.net>,
        "Schmauss, Erik" <erik.schmauss@intel.com>,
        "Verma, Vishal L" <vishal.l.verma@intel.com>,
        "Zheng, Lv" <lv.zheng@intel.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Balbir Singh <bsingharora@gmail.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jerome Glisse <jglisse@redhat.com>, John Hubbard <jhubbard@nvidia.com>,
        Len Brown <lenb@kernel.org>, Tim Chen <tim.c.chen@linux.intel.com>,
        devel@acpica.org, linux-acpi@vger.kernel.org, linux-mm@kvack.org,
        linux-nvdimm@lists.01.org
From: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Date: Fri, 22 Dec 2017 08:39:41 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.5.1
MIME-Version: 1.0
In-Reply-To: <20171214021019.13579-1-ross.zwisler@linux.intel.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Message-Id: <2d6420f7-0a95-adfe-7390-a2aea4385ab2@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3013
Lines: 67

On 12/14/2017 07:40 AM, Ross Zwisler wrote:
> ==== Quick Summary ====
> 
> Platforms exist today which have multiple types of memory attached to a
> single CPU.  These disparate memory ranges have some characteristics in
> common, such as CPU cache coherence, but they can have wide ranges of
> performance both in terms of latency and bandwidth.

Right.

> 
> For example, consider a system that contains persistent memory, standard
> DDR memory and High Bandwidth Memory (HBM), all attached to the same CPU.
> There could potentially be an order of magnitude or more difference in
> performance between the slowest and fastest memory attached to that CPU.

Right.

> 
> With the current Linux code NUMA nodes are CPU-centric, so all the memory
> attached to a given CPU will be lumped into the same NUMA node.  This makes
> it very difficult for userspace applications to understand the performance
> of different memory ranges on a given CPU.

Right but that might require fundamental changes to the NUMA representation.
Plugging those memory as separate NUMA nodes, identify them through sysfs
and try allocating from it through mbind() seems like a short term solution.

Though if we decide to go in this direction, sysfs interface or something
similar is required to enumerate memory properties.

> 
> We solve this issue by providing userspace with performance information on
> individual memory ranges.  This performance information is exposed via
> sysfs:
> 
>   # grep . mem_tgt2/* mem_tgt2/local_init/* 2>/dev/null
>   mem_tgt2/firmware_id:1
>   mem_tgt2/is_cached:0
>   mem_tgt2/local_init/read_bw_MBps:40960
>   mem_tgt2/local_init/read_lat_nsec:50
>   mem_tgt2/local_init/write_bw_MBps:40960
>   mem_tgt2/local_init/write_lat_nsec:50

I might have missed discussions from earlier versions, why we have this
kind of a "source --> target" model ? We will enlist properties for all
possible "source --> target" on the system ? Right now it shows only
bandwidth and latency properties, can it accommodate other properties
as well in future ?

> 
> This allows applications to easily find the memory that they want to use.
> We expect that the existing NUMA APIs will be enhanced to use this new
> information so that applications can continue to use them to select their
> desired memory.

I had presented a proposal for NUMA redesign in the Plumbers Conference this
year where various memory devices with different kind of memory attributes
can be represented in the kernel and be used explicitly from the user space.
Here is the link to the proposal if you feel interested. The proposal is
very intrusive and also I dont have a RFC for it yet for discussion here.

https://linuxplumbersconf.org/2017/ocw//system/presentations/4656/original/Hierarchical_NUMA_Design_Plumbers_2017.pdf

Problem is, designing the sysfs interface for memory attribute detection
from user space without first thinking about redesigning the NUMA for
heterogeneous memory may not be a good idea. Will look into this further.