Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Alistair Popple <apopple@nvidia.com>
To:     Wei Xu <weixugc@google.com>
Cc:     Yang Shi <shy828301@gmail.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Huang Ying <ying.huang@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Linux MM <linux-mm@kvack.org>,
        Greg Thelen <gthelen@google.com>,
        "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Michal Hocko <mhocko@kernel.org>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Feng Tang <feng.tang@intel.com>, Jonathan.Cameron@huawei.com,
        Tim Chen <tim.c.chen@linux.intel.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces
Date:   Fri, 06 May 2022 10:01:21 +1000
References: <CAAPL-u9sVx94ACSuCVN8V0tKp+AMxiY89cro0japtyB=xNfNBw@mail.gmail.com>
 <CAHbLzkq1YXXLMiREpGnzhJjPssu4WpSsnkTmrLJ=hAEhZVUr9w@mail.gmail.com>
 <CAAPL-u-r2Pc_MaHQmKKNH_icAa_fH1COWb5qSPpr8xffREQ_cQ@mail.gmail.com>
User-agent: mu4e 1.6.9; emacs 27.1
In-reply-to: <CAAPL-u-r2Pc_MaHQmKKNH_icAa_fH1COWb5qSPpr8xffREQ_cQ@mail.gmail.com>
Message-ID: <87tua3h5r1.fsf@nvdebian.thelocal>
Content-Type: multipart/mixed; boundary="=-=-="
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?otmS0U/7arAuzM6lDGyf1O6foVAF878n0JofGB/2pkRDx5Q5+cUHxuDDitJV?=
 =?us-ascii?Q?Yu+ZuI6c1+EAFUs2rUQ6auJVwH71NyG+brbZ1yQr4AEP44ZPzUwyFxrNSahw?=
 =?us-ascii?Q?itpbpZWPoD8Tpjm2f+7YrOhmC4ZhwAV4aU7fObajYF212KdcDIT+Ko8UYDrK?=
 =?us-ascii?Q?TCjnrQ1QZPOGT9+cCS3KJzWQRr/MIW5a5YMfTndQsg9MpZPDvkF6ob+y93QF?=
 =?us-ascii?Q?pTfL9RzUeVyd3UuzufenUp8BUabaS04xIOG3N0jyuX0x3NJFBrPE+zYhXCdp?=
 =?us-ascii?Q?VkU3tikr5oIuWns58Wkw5ErEpzruQbovfibJjt8tt35aXIhJyOUjfCODNxfP?=
 =?us-ascii?Q?z4w5ngOz6bOhzzHvmwyp9j/Z2PJ/PnV9y2sljAUWnNkL73ZukriYinaEiYfN?=
 =?us-ascii?Q?2isTkZ9ffiLOP/36l5hmYw3QWlu2yrtBpZkJYZ1KaZgP5LrjB4oPtI0IsWNR?=
 =?us-ascii?Q?asmLRCzasBjtSB6pK0C7iasQLNNf2ZTT5ejpxYJT9u83/GX75VsfktGHYvUA?=
 =?us-ascii?Q?ni6agcHPA5hETp7zD/Of5Y360joifdkRcsZLqL4Y2XZccnw2GMeerrKsAIOx?=
 =?us-ascii?Q?qXg316c/Xla6H5IcmbJRaxgDxKS0qKBzuyCnueDXHeyA9RMriEDZ6iMs3bc5?=
 =?us-ascii?Q?E1icqHsbPDzunr9G7nKAW31waKFiJxRdX6CY8t+D+tvUeI86B0tTV+1sNJ6U?=
 =?us-ascii?Q?YhSfg4kgfntWMibAp7NjZ0hhj1v54zSjFZMjOB2KE76PnNAsdmSfEsom5NON?=
 =?us-ascii?Q?lV43J1w0L/zEZF9E49bjrr751zwVyzv9kf2SlbqMta109ulLoL8bue1OUhDh?=
 =?us-ascii?Q?EKvCBqaVa2KjZM4Ntqfupd4CZZ8/Rvs3Jvk0UIfq0zgI1dqmzka5UGdqSKjt?=
 =?us-ascii?Q?VbsFYK9FmgJtSeXJPzbHnzPCneqUMpUZxuXQ368s0aMzN7YkIryuvBbisaZn?=
 =?us-ascii?Q?0ibciL6O6RV010THr9E/UhUcd9zmC6uz2GopJAn8nMHHNgpJ4zV476P04yPx?=
 =?us-ascii?Q?kRZE0tUuTmqr+VvfVN6jlK0Bvmr2wCSYBQFQOp4CZ93UAbZOLvsdOkz+ebhj?=
 =?us-ascii?Q?BOoRe6LGcXTC15yKoPSP0S1mqnbaZoxx/ugrqsC3pvFjMiIjPVBDrVC2N72L?=
 =?us-ascii?Q?oRezQ1bUcB4c42eR+57A5UGNn5C4mqS5lZR2htNUh79zB34fsaSaZ8rj9P5f?=
 =?us-ascii?Q?4A43GVf/qVYh5X1vMpoEKdek8jtK+TudKVwDwrXYONRY+pkTcX86y5e0fK3n?=
 =?us-ascii?Q?JKOPdz3tec7dBMgyWVWNAln6zMYZo2gHgOzwY5QmkpKqkFVChrEyE4gWysdp?=
 =?us-ascii?Q?w22BiN/Z68NwpzAEsVBDyXozU555xwsEzrVrdrUiFbsK8FVro3cMJNx0dIAT?=
 =?us-ascii?Q?uV6VP9vldRPEuVhqr+bgaHZw9b6CPZYqAY9wkPJiVQYm292YDEJJmwQVcorD?=
 =?us-ascii?Q?xBM/wGcTopw7o7N8cwAHFul3NcFSFEkrSI1uOAbfw0QlAVw29shEKen/yzp8?=
 =?us-ascii?Q?IX8GW4CJf3PMmJbLv2f2pQVKH4/epXEFh/sK6mcozRC+1qrCrY/yhPhJFreD?=
 =?us-ascii?Q?bxweDXm1thQ2y+HayIblAIOZZzwh+GyqoFzJvxS8xZ3XtcBttc4DNvWmkYgb?=
 =?us-ascii?Q?LaFfFUuC/a6pBLWalgQn9UYxijfhNAEHIK0axHd0/X3Dz1t+XBoG2TC5+kcx?=
 =?us-ascii?Q?IWJVv4xCg9tu46TtPt1jzl7JKJDQTl5143OchJ/f+OhlQgVy7yWlSN/cI8F0?=
 =?us-ascii?Q?nyNA5f6kZg=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cdc6415b-d7ae-453d-226b-08da2ef6206b
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2022 00:19:47.1445
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2QpUzU4wWxfkMzI4EfIdv98F9oDVVV6XvuGE26/y5jiuYMeDss5uj+QrLZRGx9cOOcjPxQanhawAMO830WuiMw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1196
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

--=-=-=
Content-Type: text/plain
Content-Disposition: inline

Wei Xu <weixugc@google.com> writes:

[...]

>> >
>> >
>> > Tiering Hierarchy Initialization
>> > `=============================='
>> >
>> > By default, all memory nodes are in the top tier (N_TOPTIER_MEMORY).
>> >
>> > A device driver can remove its memory nodes from the top tier, e.g.
>> > a dax driver can remove PMEM nodes from the top tier.
>>
>> With the topology built by firmware we should not need this.

I agree that in an ideal world the hierarchy should be built by firmware based
on something like the HMAT. But I also think being able to override this will be
useful in getting there. Therefore a way of overriding the generated hierarchy
would be good, either via sysfs or kernel boot parameter if we don't want to
commit to a particular user interface now.

However I'm less sure letting device-drivers override this is a good idea. How
for example would a GPU driver make sure it's node is in the top tier? By moving
every node that the driver does not know about out of N_TOPTIER_MEMORY? That
could get messy if say there were two drivers both of which wanted their node to
be in the top tier.

> I agree. But before we have such a firmware, the kernel needs to do
> its best to initialize memory tiers.
>
> Given that we know PMEM is slower than DRAM, but a dax device might
> not be PMEM, a better place to set the tier for PMEM nodes can be the
> ACPI code, e.g. acpi_numa_memory_affinity_init() where we can examine
> the ACPI_SRAT_MEM_NON_VOLATILE bit.
>
>> >
>> > The kernel builds the memory tiering hierarchy and per-node demotion
>> > order tier-by-tier starting from N_TOPTIER_MEMORY.  For a node N, the
>> > best distance nodes in the next lower tier are assigned to
>> > node_demotion[N].preferred and all the nodes in the next lower tier
>> > are assigned to node_demotion[N].allowed.
>>
>> I'm not sure whether it should be allowed to demote to multiple lower
>> tiers. But it is totally fine to *NOT* allow it at the moment. Once we
>> figure out a good way to define demotion targets, it could be extended
>> to support this easily.
>
> You mean to only support MAX_TIERS=2 for now.  I am fine with that.
> There can be systems with 3 tiers, e.g. GPU -> DRAM -> PMEM, but it is
> not clear yet whether we want to enable transparent memory tiering to
> all the 3 tiers on such systems.

At some point I think we will need to deal with 3 tiers but I'd be ok with
limiting it to 2 for now if it makes things simpler.

- Alistair

>> >
>> > node_demotion[N].preferred can be empty if no preferred demotion node
>> > is available for node N.
>> >
>> > If the userspace overrides the tiers via the memory_tiers sysfs
>> > interface, the kernel then only rebuilds the per-node demotion order
>> > accordingly.
>> >
>> > Memory tiering hierarchy is rebuilt upon hot-add or hot-remove of a
>> > memory node, but is NOT rebuilt upon hot-add or hot-remove of a CPU
>> > node.
>> >
>> >
>> > Memory Allocation for Demotion
>> > `============================'
>> >
>> > When allocating a new demotion target page, both a preferred node
>> > and the allowed nodemask are provided to the allocation function.
>> > The default kernel allocation fallback order is used to allocate the
>> > page from the specified node and nodemask.
>> >
>> > The memopolicy of cpuset, vma and owner task of the source page can
>> > be set to refine the demotion nodemask, e.g. to prevent demotion or
>> > select a particular allowed node as the demotion target.
>> >
>> >
>> > Examples
>> > `======'
>> >
>> > * Example 1:
>> >   Node 0 & 1 are DRAM nodes, node 2 & 3 are PMEM nodes.
>> >
>> >   Node 0 has node 2 as the preferred demotion target and can also
>> >   fallback demotion to node 3.
>> >
>> >   Node 1 has node 3 as the preferred demotion target and can also
>> >   fallback demotion to node 2.
>> >
>> >   Set mempolicy to prevent cross-socket demotion and memory access,
>> >   e.g. cpuset.mems=0,2
>> >
>> > node distances:
>> > node   0    1    2    3
>> >    0  10   20   30   40
>> >    1  20   10   40   30
>> >    2  30   40   10   40
>> >    3  40   30   40   10
>> >
>> > /sys/devices/system/node/memory_tiers
>> > 0-1
>> > 2-3
>> >
>> > N_TOPTIER_MEMORY: 0-1
>> >
>> > node_demotion[]:
>> >   0: [2], [2-3]
>> >   1: [3], [2-3]
>> >   2: [],  []
>> >   3: [],  []
>> >
>> > * Example 2:
>> >   Node 0 & 1 are DRAM nodes.
>> >   Node 2 is a PMEM node and closer to node 0.
>> >
>> >   Node 0 has node 2 as the preferred and only demotion target.
>> >
>> >   Node 1 has no preferred demotion target, but can still demote
>> >   to node 2.
>> >
>> >   Set mempolicy to prevent cross-socket demotion and memory access,
>> >   e.g. cpuset.mems=0,2
>> >
>> > node distances:
>> > node   0    1    2
>> >    0  10   20   30
>> >    1  20   10   40
>> >    2  30   40   10
>> >
>> > /sys/devices/system/node/memory_tiers
>> > 0-1
>> > 2
>> >
>> > N_TOPTIER_MEMORY: 0-1
>> >
>> > node_demotion[]:
>> >   0: [2], [2]
>> >   1: [],  [2]
>> >   2: [],  []
>> >
>> >
>> > * Example 3:
>> >   Node 0 & 1 are DRAM nodes.
>> >   Node 2 is a PMEM node and has the same distance to node 0 & 1.
>> >
>> >   Node 0 has node 2 as the preferred and only demotion target.
>> >
>> >   Node 1 has node 2 as the preferred and only demotion target.
>> >
>> > node distances:
>> > node   0    1    2
>> >    0  10   20   30
>> >    1  20   10   30
>> >    2  30   30   10
>> >
>> > /sys/devices/system/node/memory_tiers
>> > 0-1
>> > 2
>> >
>> > N_TOPTIER_MEMORY: 0-1
>> >
>> > node_demotion[]:
>> >   0: [2], [2]
>> >   1: [2], [2]
>> >   2: [],  []
>> >
>> >
>> > * Example 4:
>> >   Node 0 & 1 are DRAM nodes, Node 2 is a memory-only DRAM node.
>> >
>> >   All nodes are top-tier.
>> >
>> > node distances:
>> > node   0    1    2
>> >    0  10   20   30
>> >    1  20   10   30
>> >    2  30   30   10
>> >
>> > /sys/devices/system/node/memory_tiers
>> > 0-2
>> >
>> > N_TOPTIER_MEMORY: 0-2
>> >
>> > node_demotion[]:
>> >   0: [],  []
>> >   1: [],  []
>> >   2: [],  []
>> >
>> >
>> > * Example 5:
>> >   Node 0 is a DRAM node with CPU.
>> >   Node 1 is a HBM node.
>> >   Node 2 is a PMEM node.
>> >
>> >   With userspace override, node 1 is the top tier and has node 0 as
>> >   the preferred and only demotion target.
>> >
>> >   Node 0 is in the second tier, tier 1, and has node 2 as the
>> >   preferred and only demotion target.
>> >
>> >   Node 2 is in the lowest tier, tier 2, and has no demotion targets.
>> >
>> > node distances:
>> > node   0    1    2
>> >    0  10   21   30
>> >    1  21   10   40
>> >    2  30   40   10
>> >
>> > /sys/devices/system/node/memory_tiers (userspace override)
>> > 1
>> > 0
>> > 2
>> >
>> > N_TOPTIER_MEMORY: 1
>> >
>> > node_demotion[]:
>> >   0: [2], [2]
>> >   1: [0], [0]
>> >   2: [],  []
>> >
>> > -- Wei

--=-=-=--