The current kernel has the basic memory tiering support: Inactive
pages on a higher tier NUMA node can be migrated (demoted) to a lower
tier NUMA node to make room for new allocations on the higher tier
NUMA node. Frequently accessed pages on a lower tier NUMA node can be
migrated (promoted) to a higher tier NUMA node to improve the
performance.
In the current kernel, memory tiers are defined implicitly via a
demotion path relationship between NUMA nodes, which is created during
the kernel initialization and updated when a NUMA node is hot-added or
hot-removed. The current implementation puts all nodes with CPU into
the top tier, and builds the tier hierarchy tier-by-tier by establishing
the per-node demotion targets based on the distances between nodes.
This current memory tier kernel interface needs to be improved for
several important use cases:
* The current tier initialization code always initializes
each memory-only NUMA node into a lower tier. But a memory-only
NUMA node may have a high performance memory device (e.g. a DRAM
device attached via CXL.mem or a DRAM-backed memory-only node on
a virtual machine) and should be put into a higher tier.
* The current tier hierarchy always puts CPU nodes into the top
tier. But on a system with HBM (e.g. GPU memory) devices, these
memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes
with CPUs are better to be placed into the next lower tier.
* Also because the current tier hierarchy always puts CPU nodes
into the top tier, when a CPU is hot-added (or hot-removed) and
triggers a memory node from CPU-less into a CPU node (or vice
versa), the memory tier hierarchy gets changed, even though no
memory node is added or removed. This can make the tier
hierarchy unstable and make it difficult to support tier-based
memory accounting.
* A higher tier node can only be demoted to selected nodes on the
next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order
does not work in all use cases (e.g. some use cases may want to
allow cross-socket demotion to another node in the same demotion
tier as a fallback when the preferred demotion node is out of
space), and has resulted in the feature request for an interface to
override the system-wide, per-node demotion order from the
userspace. This demotion order is also inconsistent with the page
allocation fallback order when all the nodes in a higher tier are
out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.
* There are no interfaces for the userspace to learn about the memory
tier hierarchy in order to optimize its memory allocations.
This patch series make the creation of memory tiers explicit under
the control of userspace or device driver.
Memory Tier Initialization
==========================
By default, all memory nodes are assigned to the default tier (1).
The default tier device has a rank value (200).
A device driver can move up or down its memory nodes from the default
tier. For example, PMEM can move down its memory nodes below the
default tier, whereas GPU can move up its memory nodes above the
default tier.
The kernel initialization code makes the decision on which exact tier
a memory node should be assigned to based on the requests from the
device drivers as well as the memory device hardware information
provided by the firmware.
Hot-adding/removing CPUs doesn't affect memory tier hierarchy.
Memory Allocation for Demotion
==============================
This patch series keep the demotion target page allocation logic same.
The demotion page allocation pick the closest NUMA node in the
next lower tier to the current NUMA node allocating pages from.
This will be later improved to use the same page allocation strategy
using fallback list.
Sysfs Interface:
=======================
Listing current list of memory tiers and rank details:
:/sys/devices/system/memtier$ ls
default_tier max_tier memtier1 power uevent
:/sys/devices/system/memtier$ cat default_tier
memtier1
:/sys/devices/system/memtier$ cat max_tier
3
:/sys/devices/system/memtier$
Per node memory tier details:
For a cpu only NUMA node:
:/sys/devices/system/node# cat node0/memtier
:/sys/devices/system/node# echo 1 > node0/memtier
:/sys/devices/system/node# cat node0/memtier
:/sys/devices/system/node#
For a NUMA node with memory:
:/sys/devices/system/node# cat node1/memtier
1
:/sys/devices/system/node# ls ../memtier/
default_tier max_tier memtier1 power uevent
:/sys/devices/system/node# echo 2 > node1/memtier
:/sys/devices/system/node#
:/sys/devices/system/node# ls ../memtier/
default_tier max_tier memtier1 memtier2 power uevent
:/sys/devices/system/node# cat node1/memtier
2
:/sys/devices/system/node#
:/sys/devices/system/node# cat ../memtier/memtier2/rank
100
:/sys/devices/system/node#
:/sys/devices/system/node# cat ../memtier/memtier1/rank
200
:/sys/devices/system/node#
Removing a NUMA node from demotion:
:/sys/devices/system/node# cat node1/memtier
2
:/sys/devices/system/node# echo none > node1/memtier
:/sys/devices/system/node#
:/sys/devices/system/node# cat node1/memtier
:/sys/devices/system/node#
:/sys/devices/system/node# ls ../memtier/
default_tier max_tier memtier1 power uevent
:/sys/devices/system/node#
The above also resulted in removal of memtier2 which was created in the earlier step.
Changes from v5:
* Remove patch supporting N_MEMORY node removal from memory tiers. memory tiers
are going to be used for features other than demotion. Hence keep all N_MEMORY
nodes in memory tiers irrespective of whether they want to participate in promotion or demotion.
* Add NODE_DATA->memtier
* Rearrage patches to add sysfs files later.
* Add support to create memory tiers from userspace.
* Address other review feedback.
Changes from v4:
* Address review feedback.
* Reverse the meaning of "rank": higher rank value means higher tier.
* Add "/sys/devices/system/memtier/default_tier".
* Add node_is_toptier
v4:
Add support for explicit memory tiers and ranks.
v3:
- Modify patch 1 subject to make it more specific
- Remove /sys/kernel/mm/numa/demotion_targets interface, use
/sys/devices/system/node/demotion_targets instead and make
it writable to override node_states[N_DEMOTION_TARGETS].
- Add support to view per node demotion targets via sysfs
v2:
In v1, only 1st patch of this patch series was sent, which was
implemented to avoid some of the limitations on the demotion
target sharing, however for certain numa topology, the demotion
targets found by that patch was not most optimal, so 1st patch
in this series is modified according to suggestions from Huang
and Baolin. Different examples of demotion list comparasion
between existing implementation and changed implementation can
be found in the commit message of 1st patch.
Aneesh Kumar K.V (11):
mm/demotion: Add support for explicit memory tiers
mm/demotion: Move memory demotion related code
mm/demotion: Return error on write to numa_demotion sysfs
mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM
mm/demotion: Build demotion targets based on explicit memory tiers
mm/demotion: Expose memory tier details via sysfs
mm/demotion: Add per node memory tier attribute to sysfs
mm/demotion: Add support for memory tier creation from userspace
mm/demotion: Add pg_data_t member to track node memory tier details
mm/demotion: Update node_is_toptier to work with memory tiers
mm/demotion: Add sysfs ABI documentation
Jagdish Gediya (2):
mm/demotion: Demote pages according to allocation fallback order
mm/demotion: Add documentation for memory tiering
.../ABI/testing/sysfs-kernel-mm-memory-tiers | 87 ++
Documentation/admin-guide/mm/index.rst | 1 +
.../admin-guide/mm/memory-tiering.rst | 181 ++++
drivers/base/node.c | 39 +
drivers/dax/kmem.c | 4 +
include/linux/memory-tiers.h | 63 ++
include/linux/migrate.h | 15 -
include/linux/mmzone.h | 3 +
include/linux/node.h | 5 -
mm/Kconfig | 3 +
mm/Makefile | 1 +
mm/huge_memory.c | 1 +
mm/memory-tiers.c | 888 ++++++++++++++++++
mm/migrate.c | 453 +--------
mm/mprotect.c | 1 +
mm/vmscan.c | 57 +-
mm/vmstat.c | 4 -
17 files changed, 1316 insertions(+), 490 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
create mode 100644 Documentation/admin-guide/mm/memory-tiering.rst
create mode 100644 include/linux/memory-tiers.h
create mode 100644 mm/memory-tiers.c
--
2.36.1
This patch adds /sys/devices/system/memtier/ where all memory tier
related details can be found. All created memory tiers will be
listed there as /sys/devices/system/memtier/memtierN/
The nodes which are part of a specific memory tier can be listed
via /sys/devices/system/memtier/memtierN/nodelist
The rank value of a memory tier can be listed via
via /sys/devices/system/memtier/memtierN/rank
/sys/devices/system/memtier/max_tier shows the maximum number of memory
tiers that can be created.
/sys/devices/system/memtier/default_tier shows the memory tier to which
NUMA nodes get added by default if not assigned a specific memory tier.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
mm/memory-tiers.c | 99 +++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 95 insertions(+), 4 deletions(-)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 2f116912de43..51210f5efc1f 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -11,8 +11,8 @@
struct memory_tier {
struct list_head list;
+ struct device dev;
nodemask_t nodelist;
- int id;
int rank;
};
@@ -20,6 +20,7 @@ struct demotion_nodes {
nodemask_t preferred;
};
+#define to_memory_tier(device) container_of(device, struct memory_tier, dev)
static void establish_migration_targets(void);
static DEFINE_MUTEX(memory_tier_lock);
static LIST_HEAD(memory_tiers);
@@ -86,6 +87,52 @@ static LIST_HEAD(memory_tiers);
*/
static struct demotion_nodes *node_demotion __read_mostly;
+static struct bus_type memory_tier_subsys = {
+ .name = "memtier",
+ .dev_name = "memtier",
+};
+
+static ssize_t nodelist_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct memory_tier *memtier = to_memory_tier(dev);
+
+ return sysfs_emit(buf, "%*pbl\n",
+ nodemask_pr_args(&memtier->nodelist));
+}
+static DEVICE_ATTR_RO(nodelist);
+
+static ssize_t rank_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct memory_tier *memtier = to_memory_tier(dev);
+
+ return sysfs_emit(buf, "%d\n", memtier->rank);
+}
+static DEVICE_ATTR_RO(rank);
+
+static struct attribute *memory_tier_dev_attrs[] = {
+ &dev_attr_nodelist.attr,
+ &dev_attr_rank.attr,
+ NULL
+};
+
+static const struct attribute_group memory_tier_dev_group = {
+ .attrs = memory_tier_dev_attrs,
+};
+
+static const struct attribute_group *memory_tier_dev_groups[] = {
+ &memory_tier_dev_group,
+ NULL
+};
+
+static void memory_tier_device_release(struct device *dev)
+{
+ struct memory_tier *tier = to_memory_tier(dev);
+
+ kfree(tier);
+}
+
/*
* Keep it simple by having direct mapping between
* tier index and rank value.
@@ -121,6 +168,7 @@ static void insert_memory_tier(struct memory_tier *memtier)
static struct memory_tier *register_memory_tier(unsigned int tier,
unsigned int rank)
{
+ int error;
struct memory_tier *memtier;
if (tier >= MAX_MEMORY_TIERS)
@@ -130,11 +178,20 @@ static struct memory_tier *register_memory_tier(unsigned int tier,
if (!memtier)
return ERR_PTR(-ENOMEM);
- memtier->id = tier;
+ memtier->dev.id = tier;
memtier->rank = rank;
+ memtier->dev.bus = &memory_tier_subsys;
+ memtier->dev.release = memory_tier_device_release;
+ memtier->dev.groups = memory_tier_dev_groups;
insert_memory_tier(memtier);
+ error = device_register(&memtier->dev);
+ if (error) {
+ list_del(&memtier->list);
+ put_device(&memtier->dev);
+ return ERR_PTR(error);
+ }
return memtier;
}
@@ -154,7 +211,7 @@ static struct memory_tier *__get_memory_tier_from_id(int id)
struct memory_tier *memtier;
list_for_each_entry(memtier, &memory_tiers, list) {
- if (memtier->id == id)
+ if (memtier->dev.id == id)
return memtier;
}
return NULL;
@@ -199,7 +256,7 @@ int node_create_and_set_memory_tier(int node, int tier)
goto out;
}
- if (current_tier->id == tier)
+ if (current_tier->dev.id == tier)
goto out;
node_clear(node, current_tier->nodelist);
@@ -435,10 +492,44 @@ static void __init migrate_on_reclaim_init(void)
hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
}
+static ssize_t
+max_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ return sysfs_emit(buf, "%d\n", MAX_MEMORY_TIERS);
+}
+static DEVICE_ATTR_RO(max_tier);
+
+static ssize_t
+default_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ return sysfs_emit(buf, "memtier%d\n", DEFAULT_MEMORY_TIER);
+}
+static DEVICE_ATTR_RO(default_tier);
+
+static struct attribute *memory_tier_attrs[] = {
+ &dev_attr_max_tier.attr,
+ &dev_attr_default_tier.attr,
+ NULL
+};
+
+static const struct attribute_group memory_tier_attr_group = {
+ .attrs = memory_tier_attrs,
+};
+
+static const struct attribute_group *memory_tier_attr_groups[] = {
+ &memory_tier_attr_group,
+ NULL,
+};
+
static int __init memory_tier_init(void)
{
+ int ret;
struct memory_tier *memtier;
+ ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups);
+ if (ret)
+ pr_err("%s() failed to register subsystem: %d\n", __func__, ret);
+
/*
* Register only default memory tier to hide all empty
* memory tier from sysfs.
--
2.36.1
Add support to modify the memory tier for a NUMA node.
/sys/devices/system/node/nodeN/memtier
where N = node id
When read, It list the memory tier that the node belongs to.
When written, the kernel moves the node into the specified
memory tier, the tier assignment of all other nodes are not
affected.
If the memory tier does not exist an error is returned.
Suggested-by: Wei Xu <[email protected]>
Signed-off-by: Jagdish Gediya <[email protected]>
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
drivers/base/node.c | 39 ++++++++++++++++++++++++++++++++
include/linux/memory-tiers.h | 3 +++
mm/memory-tiers.c | 44 ++++++++++++++++++++++++++++++++++++
3 files changed, 86 insertions(+)
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 0ac6376ef7a1..599ed64d910f 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -20,6 +20,7 @@
#include <linux/pm_runtime.h>
#include <linux/swap.h>
#include <linux/slab.h>
+#include <linux/memory-tiers.h>
static struct bus_type node_subsys = {
.name = "node",
@@ -560,11 +561,49 @@ static ssize_t node_read_distance(struct device *dev,
}
static DEVICE_ATTR(distance, 0444, node_read_distance, NULL);
+#ifdef CONFIG_TIERED_MEMORY
+static ssize_t memtier_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int node = dev->id;
+ int tier_index = node_get_memory_tier_id(node);
+
+ if (tier_index != -1)
+ return sysfs_emit(buf, "%d\n", tier_index);
+ return 0;
+}
+
+static ssize_t memtier_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ unsigned long tier;
+ int node = dev->id;
+ int ret;
+
+ ret = kstrtoul(buf, 10, &tier);
+ if (ret)
+ return ret;
+
+ ret = node_reset_memory_tier(node, tier);
+ if (ret)
+ return ret;
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(memtier);
+#endif
+
static struct attribute *node_dev_attrs[] = {
&dev_attr_meminfo.attr,
&dev_attr_numastat.attr,
&dev_attr_distance.attr,
&dev_attr_vmstat.attr,
+#ifdef CONFIG_TIERED_MEMORY
+ &dev_attr_memtier.attr,
+#endif
NULL
};
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 18dd1ab7b96e..e70f0040d845 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -20,6 +20,9 @@
extern bool numa_demotion_enabled;
int node_create_and_set_memory_tier(int node, int tier);
int next_demotion_node(int node);
+int node_set_memory_tier(int node, int tier);
+int node_get_memory_tier_id(int node);
+int node_reset_memory_tier(int node, int tier);
#else
#define numa_demotion_enabled false
static inline int next_demotion_node(int node)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 51210f5efc1f..7bfdfac4d43e 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -313,6 +313,50 @@ int node_set_memory_tier(int node, int tier)
return ret;
}
+int node_get_memory_tier_id(int node)
+{
+ int tier = -1;
+ struct memory_tier *memtier;
+ /*
+ * Make sure memory tier is not unregistered
+ * while it is being read.
+ */
+ mutex_lock(&memory_tier_lock);
+ memtier = __node_get_memory_tier(node);
+ if (memtier)
+ tier = memtier->dev.id;
+ mutex_unlock(&memory_tier_lock);
+
+ return tier;
+}
+
+int node_reset_memory_tier(int node, int tier)
+{
+ struct memory_tier *current_tier;
+ int ret = 0;
+
+ mutex_lock(&memory_tier_lock);
+
+ current_tier = __node_get_memory_tier(node);
+ if (!current_tier || current_tier->dev.id == tier)
+ goto out;
+
+ node_clear(node, current_tier->nodelist);
+
+ ret = __node_set_memory_tier(node, tier);
+ if (ret) {
+ /* reset it back to older tier */
+ node_set(node, current_tier->nodelist);
+ goto out;
+ }
+
+ establish_migration_targets();
+out:
+ mutex_unlock(&memory_tier_lock);
+
+ return ret;
+}
+
/**
* next_demotion_node() - Get the next node in the demotion path
* @node: The starting node to lookup the next node
--
2.36.1
In the current kernel, memory tiers are defined implicitly via a
demotion path relationship between NUMA nodes, which is created
during the kernel initialization and updated when a NUMA node is
hot-added or hot-removed. The current implementation puts all
nodes with CPU into the top tier, and builds the tier hierarchy
tier-by-tier by establishing the per-node demotion targets based
on the distances between nodes.
This current memory tier kernel interface needs to be improved for
several important use cases,
The current tier initialization code always initializes
each memory-only NUMA node into a lower tier. But a memory-only
NUMA node may have a high performance memory device (e.g. a DRAM
device attached via CXL.mem or a DRAM-backed memory-only node on
a virtual machine) and should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top
tier. But on a system with HBM or GPU devices, the
memory-only NUMA nodes mapping these devices should be in the
top tier, and DRAM nodes with CPUs are better to be placed into the
next lower tier.
With current kernel higher tier node can only be demoted to selected nodes on the
next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, hard-coded demotion order
does not work in all use cases (e.g. some use cases may want to
allow cross-socket demotion to another node in the same demotion
tier as a fallback when the preferred demotion node is out of
space), This demotion order is also inconsistent with the page
allocation fallback order when all the nodes in a higher tier are
out of space: The page allocation can fall back to any node from
any lower tier, whereas the demotion order doesn't allow that.
The current kernel also don't provide any interfaces for the
userspace to learn about the memory tier hierarchy in order to
optimize its memory allocations.
This patch series address the above by defining memory tiers explicitly.
This patch introduce explicity memory tiers with ranks. The rank
value of a memory tier is used to derive the demotion order between
NUMA nodes. The memory tiers present in a system can be found at
"Rank" is an opaque value. Its absolute value doesn't have any
special meaning. But the rank values of different memtiers can be
compared with each other to determine the memory tier order.
For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and
their rank values are 300, 200, 100, then the memory tier order is:
memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier
and memtier2 is the lowest tier.
The rank value of each memtier should be unique.
A higher rank memory tier will appear first in the demotion order
than a lower rank memory tier. ie. while reclaim we choose a node
in higher rank memory tier to demote pages to as compared to a node
in a lower rank memory tier.
This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2)
which are created by different kernel subsystems. The default memory
tier created by the kernel is memtier1. Once created these memory tiers
are not destroyed even if they don't have any NUMA nodes assigned to
them.
This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
[1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
/sys/devices/system/memtier/memtierN/
The nodes which are part of a specific memory tier can be listed
via
/sys/devices/system/memtier/memtierN/nodelist
Suggested-by: Wei Xu <[email protected]>
Signed-off-by: Jagdish Gediya <[email protected]>
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/memory-tiers.h | 20 ++++++++
mm/Kconfig | 3 ++
mm/Makefile | 1 +
mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++
4 files changed, 113 insertions(+)
create mode 100644 include/linux/memory-tiers.h
create mode 100644 mm/memory-tiers.c
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
new file mode 100644
index 000000000000..e17f6b4ee177
--- /dev/null
+++ b/include/linux/memory-tiers.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+#ifdef CONFIG_TIERED_MEMORY
+
+#define MEMORY_TIER_HBM_GPU 0
+#define MEMORY_TIER_DRAM 1
+#define MEMORY_TIER_PMEM 2
+
+#define MEMORY_RANK_HBM_GPU 300
+#define MEMORY_RANK_DRAM 200
+#define MEMORY_RANK_PMEM 100
+
+#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM
+#define MAX_MEMORY_TIERS 3
+
+#endif /* CONFIG_TIERED_MEMORY */
+
+#endif
diff --git a/mm/Kconfig b/mm/Kconfig
index 169e64192e48..bb5aa585ab41 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
config ARCH_ENABLE_THP_MIGRATION
bool
+config TIERED_MEMORY
+ def_bool NUMA
+
config HUGETLB_PAGE_SIZE_VARIABLE
def_bool n
help
diff --git a/mm/Makefile b/mm/Makefile
index 6f9ffa968a1a..482557fbc9d1 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o
obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
new file mode 100644
index 000000000000..d9fa955f208e
--- /dev/null
+++ b/mm/memory-tiers.c
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+#include <linux/nodemask.h>
+#include <linux/slab.h>
+#include <linux/memory-tiers.h>
+
+struct memory_tier {
+ struct list_head list;
+ nodemask_t nodelist;
+ int id;
+ int rank;
+};
+
+static DEFINE_MUTEX(memory_tier_lock);
+static LIST_HEAD(memory_tiers);
+
+/*
+ * Keep it simple by having direct mapping between
+ * tier index and rank value.
+ */
+static inline int get_rank_from_tier(unsigned int tier)
+{
+ switch (tier) {
+ case MEMORY_TIER_HBM_GPU:
+ return MEMORY_RANK_HBM_GPU;
+ case MEMORY_TIER_DRAM:
+ return MEMORY_RANK_DRAM;
+ case MEMORY_TIER_PMEM:
+ return MEMORY_RANK_PMEM;
+ }
+ return -1;
+}
+
+static void insert_memory_tier(struct memory_tier *memtier)
+{
+ struct list_head *ent;
+ struct memory_tier *tmp_memtier;
+
+ list_for_each(ent, &memory_tiers) {
+ tmp_memtier = list_entry(ent, struct memory_tier, list);
+ if (tmp_memtier->rank < memtier->rank) {
+ list_add_tail(&memtier->list, ent);
+ return;
+ }
+ }
+ list_add_tail(&memtier->list, &memory_tiers);
+}
+
+static struct memory_tier *register_memory_tier(unsigned int tier,
+ unsigned int rank)
+{
+ struct memory_tier *memtier;
+
+ if (tier >= MAX_MEMORY_TIERS)
+ return ERR_PTR(-EINVAL);
+
+ memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
+ if (!memtier)
+ return ERR_PTR(-ENOMEM);
+
+ memtier->id = tier;
+ memtier->rank = rank;
+
+ insert_memory_tier(memtier);
+
+ return memtier;
+}
+
+static int __init memory_tier_init(void)
+{
+ struct memory_tier *memtier;
+
+ /*
+ * Register only default memory tier to hide all empty
+ * memory tier from sysfs.
+ */
+ memtier = register_memory_tier(DEFAULT_MEMORY_TIER,
+ get_rank_from_tier(DEFAULT_MEMORY_TIER));
+
+ if (IS_ERR(memtier))
+ panic("%s() failed to register memory tier: %ld\n",
+ __func__, PTR_ERR(memtier));
+
+ /* CPU only nodes are not part of memory tiers. */
+ memtier->nodelist = node_states[N_MEMORY];
+
+ return 0;
+}
+subsys_initcall(memory_tier_init);
--
2.36.1
Also update different helpes to use NODE_DATA()->memtier. Since
node specific memtier can change based on the reassignment of
NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
needs to under an rcu read lock of memory_tier_lock.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/memory-tiers.h | 14 +++++
include/linux/mmzone.h | 3 ++
mm/memory-tiers.c | 102 ++++++++++++++++++++++++++---------
3 files changed, 94 insertions(+), 25 deletions(-)
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 52896f5970b7..53f3e4c7cba8 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -6,6 +6,9 @@
#ifdef CONFIG_TIERED_MEMORY
+#include <linux/device.h>
+#include <linux/nodemask.h>
+
#define MEMORY_TIER_HBM_GPU 0
#define MEMORY_TIER_DRAM 1
#define MEMORY_TIER_PMEM 2
@@ -18,13 +21,24 @@
#define MAX_STATIC_MEMORY_TIERS 3
#define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2)
+struct memory_tier {
+ struct list_head list;
+ struct device dev;
+ nodemask_t nodelist;
+ int rank;
+};
+
extern bool numa_demotion_enabled;
int node_create_and_set_memory_tier(int node, int tier);
int next_demotion_node(int node);
int node_set_memory_tier(int node, int tier);
int node_get_memory_tier_id(int node);
int node_reset_memory_tier(int node, int tier);
+struct memory_tier *node_get_memory_tier(int node);
+void node_put_memory_tier(struct memory_tier *memtier);
+
#else
+
#define numa_demotion_enabled false
static inline int next_demotion_node(int node)
{
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index aab70355d64f..c4fcfd2b9980 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -928,6 +928,9 @@ typedef struct pglist_data {
/* Per-node vmstats */
struct per_cpu_nodestat __percpu *per_cpu_nodestats;
atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
+#ifdef CONFIG_TIERED_MEMORY
+ struct memory_tier *memtier;
+#endif
} pg_data_t;
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index de3b7403ae6f..429aa864edb0 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -1,22 +1,14 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/types.h>
-#include <linux/device.h>
-#include <linux/nodemask.h>
#include <linux/slab.h>
#include <linux/memory-tiers.h>
#include <linux/random.h>
#include <linux/memory.h>
#include <linux/idr.h>
+#include <linux/rcupdate.h>
#include "internal.h"
-struct memory_tier {
- struct list_head list;
- struct device dev;
- nodemask_t nodelist;
- int rank;
-};
-
struct demotion_nodes {
nodemask_t preferred;
};
@@ -212,13 +204,17 @@ static void unregister_memory_tier(struct memory_tier *memtier)
static struct memory_tier *__node_get_memory_tier(int node)
{
- struct memory_tier *memtier;
+ pg_data_t *pgdat;
- list_for_each_entry(memtier, &memory_tiers, list) {
- if (node_isset(node, memtier->nodelist))
- return memtier;
- }
- return NULL;
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return NULL;
+ /*
+ * Since we hold memory_tier_lock, we can avoid
+ * RCU read locks when accessing the details. No
+ * parallel updates are possible here.
+ */
+ return pgdat->memtier;
}
static struct memory_tier *__get_memory_tier_from_id(int id)
@@ -232,6 +228,32 @@ static struct memory_tier *__get_memory_tier_from_id(int id)
return NULL;
}
+/*
+ * Called with memory_tier_lock. Hence the device references cannot
+ * be dropped during this function.
+ */
+static void memtier_node_set(int node, struct memory_tier *memtier)
+{
+ pg_data_t *pgdat;
+ struct memory_tier *current_memtier;
+
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return;
+ /*
+ * Make sure we mark the memtier NULL before we assign the new memory tier
+ * to the NUMA node. This make sure that anybody looking at NODE_DATA
+ * finds a NULL memtier or the one which is still valid.
+ */
+ current_memtier = rcu_dereference(pgdat->memtier);
+ rcu_assign_pointer(pgdat->memtier, NULL);
+ if (current_memtier)
+ node_clear(node, current_memtier->nodelist);
+ synchronize_rcu();
+ node_set(node, memtier->nodelist);
+ rcu_assign_pointer(pgdat->memtier, memtier);
+}
+
static int __node_create_and_set_memory_tier(int node, int tier)
{
int ret = 0;
@@ -252,7 +274,7 @@ static int __node_create_and_set_memory_tier(int node, int tier)
goto out;
}
}
- node_set(node, memtier->nodelist);
+ memtier_node_set(node, memtier);
out:
return ret;
}
@@ -274,12 +296,10 @@ int node_create_and_set_memory_tier(int node, int tier)
if (current_tier->dev.id == tier)
goto out;
- node_clear(node, current_tier->nodelist);
-
ret = __node_create_and_set_memory_tier(node, tier);
if (ret) {
/* reset it back to older tier */
- node_set(node, current_tier->nodelist);
+ memtier_node_set(node, current_tier);
goto out;
}
@@ -304,7 +324,7 @@ static int __node_set_memory_tier(int node, int tier)
ret = -EINVAL;
goto out;
}
- node_set(node, memtier->nodelist);
+ memtier_node_set(node, memtier);
out:
return ret;
}
@@ -360,12 +380,10 @@ int node_reset_memory_tier(int node, int tier)
if (!current_tier || current_tier->dev.id == tier)
goto out;
- node_clear(node, current_tier->nodelist);
-
ret = __node_set_memory_tier(node, tier);
if (ret) {
/* reset it back to older tier */
- node_set(node, current_tier->nodelist);
+ memtier_node_set(node, current_tier);
goto out;
}
@@ -379,6 +397,34 @@ int node_reset_memory_tier(int node, int tier)
return ret;
}
+/*
+ * lockless access to memory tier of a NUMA node.
+ */
+struct memory_tier *node_get_memory_tier(int node)
+{
+ pg_data_t *pgdat;
+ struct memory_tier *memtier;
+
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return NULL;
+
+ rcu_read_lock();
+ memtier = rcu_dereference(pgdat->memtier);
+ if (!memtier)
+ goto out;
+
+ get_device(&memtier->dev);
+out:
+ rcu_read_unlock();
+ return memtier;
+}
+
+void node_put_memory_tier(struct memory_tier *memtier)
+{
+ put_device(&memtier->dev);
+}
+
/**
* next_demotion_node() - Get the next node in the demotion path
* @node: The starting node to lookup the next node
@@ -641,7 +687,7 @@ static const struct attribute_group *memory_tier_attr_groups[] = {
static int __init memory_tier_init(void)
{
- int ret;
+ int ret, node;
struct memory_tier *memtier;
ret = subsys_system_register(&memory_tier_subsys, memory_tier_attr_groups);
@@ -660,7 +706,13 @@ static int __init memory_tier_init(void)
__func__, PTR_ERR(memtier));
/* CPU only nodes are not part of memory tiers. */
- memtier->nodelist = node_states[N_MEMORY];
+ for_each_node_state(node, N_MEMORY) {
+ /*
+ * Should be safe to do this early in the boot.
+ */
+ NODE_DATA(node)->memtier = memtier;
+ node_set(node, memtier->nodelist);
+ }
migrate_on_reclaim_init();
return 0;
--
2.36.1
This patch switch the demotion target building logic to use memory tiers
instead of NUMA distance. All N_MEMORY NUMA nodes will be placed in the
default tier 1 and additional memory tiers will be added by drivers like
dax kmem.
This patch builds the demotion target for a NUMA node by looking at all
memory tiers below the tier to which the NUMA node belongs. The closest node
in the immediately following memory tier is used as a demotion target.
Since we are now only building demotion target for N_MEMORY NUMA nodes
the CPU hotplug calls are removed in this patch.
The rank approach allows us to keep memory tier device IDs stable even if there
is a need to change the tier ordering among different memory tiers. e.g. DRAM
nodes with CPUs will always be on memtier1, no matter how many tiers are higher
or lower than these nodes. A new memory tier can be inserted into the tier
hierarchy for a new set of nodes without affecting the node assignment of any
existing memtier, provided that there is enough gap in the rank values for the
new memtier.
The absolute value of "rank" of a memtier doesn't necessarily carry any meaning.
Its value relative to other memtiers decides the level of this memtier in the tier
hierarchy.
For now, This patch supports hardcoded rank values which are 300, 200, & 100 for
memory tiers 0,1 & 2 respectively.
Suggested-by: Wei Xu <[email protected]>
Signed-off-by: Aneesh Kumar K.V <[email protected]>
Below is the sysfs interface to read the rank values of memory tier,
/sys/devices/system/memtier/memtierN/rank
This interface is read only for now. Write support can be added when there is
a need of flexibility of more number of memory tiers(> 3) with flexibile ordering
requirement among them.
---
include/linux/memory-tiers.h | 5 +
include/linux/migrate.h | 13 --
mm/memory-tiers.c | 291 ++++++++++++++++++++++++++
mm/migrate.c | 394 -----------------------------------
mm/vmstat.c | 4 -
5 files changed, 296 insertions(+), 411 deletions(-)
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index e102ec73ab80..18dd1ab7b96e 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -19,8 +19,13 @@
extern bool numa_demotion_enabled;
int node_create_and_set_memory_tier(int node, int tier);
+int next_demotion_node(int node);
#else
#define numa_demotion_enabled false
+static inline int next_demotion_node(int node)
+{
+ return NUMA_NO_NODE;
+}
#endif /* CONFIG_TIERED_MEMORY */
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 43e737215f33..93fab62e6548 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -75,19 +75,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
#endif /* CONFIG_MIGRATION */
-#if defined(CONFIG_MIGRATION) && defined(CONFIG_NUMA)
-extern void set_migration_target_nodes(void);
-extern void migrate_on_reclaim_init(void);
-extern int next_demotion_node(int node);
-#else
-static inline void set_migration_target_nodes(void) {}
-static inline void migrate_on_reclaim_init(void) {}
-static inline int next_demotion_node(int node)
-{
- return NUMA_NO_NODE;
-}
-#endif
-
#ifdef CONFIG_COMPACTION
extern int PageMovable(struct page *page);
extern void __SetPageMovable(struct page *page, struct address_space *mapping);
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 00d393a5a628..2f116912de43 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -4,6 +4,10 @@
#include <linux/nodemask.h>
#include <linux/slab.h>
#include <linux/memory-tiers.h>
+#include <linux/random.h>
+#include <linux/memory.h>
+
+#include "internal.h"
struct memory_tier {
struct list_head list;
@@ -12,9 +16,76 @@ struct memory_tier {
int rank;
};
+struct demotion_nodes {
+ nodemask_t preferred;
+};
+
+static void establish_migration_targets(void);
static DEFINE_MUTEX(memory_tier_lock);
static LIST_HEAD(memory_tiers);
+/*
+ * node_demotion[] examples:
+ *
+ * Example 1:
+ *
+ * Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM nodes.
+ *
+ * node distances:
+ * node 0 1 2 3
+ * 0 10 20 30 40
+ * 1 20 10 40 30
+ * 2 30 40 10 40
+ * 3 40 30 40 10
+ *
+ * memory_tiers[0] = <empty>
+ * memory_tiers[1] = 0-1
+ * memory_tiers[2] = 2-3
+ *
+ * node_demotion[0].preferred = 2
+ * node_demotion[1].preferred = 3
+ * node_demotion[2].preferred = <empty>
+ * node_demotion[3].preferred = <empty>
+ *
+ * Example 2:
+ *
+ * Node 0 & 1 are CPU + DRAM nodes, node 2 is memory-only DRAM node.
+ *
+ * node distances:
+ * node 0 1 2
+ * 0 10 20 30
+ * 1 20 10 30
+ * 2 30 30 10
+ *
+ * memory_tiers[0] = <empty>
+ * memory_tiers[1] = 0-2
+ * memory_tiers[2] = <empty>
+ *
+ * node_demotion[0].preferred = <empty>
+ * node_demotion[1].preferred = <empty>
+ * node_demotion[2].preferred = <empty>
+ *
+ * Example 3:
+ *
+ * Node 0 is CPU + DRAM nodes, Node 1 is HBM node, node 2 is PMEM node.
+ *
+ * node distances:
+ * node 0 1 2
+ * 0 10 20 30
+ * 1 20 10 40
+ * 2 30 40 10
+ *
+ * memory_tiers[0] = 1
+ * memory_tiers[1] = 0
+ * memory_tiers[2] = 2
+ *
+ * node_demotion[0].preferred = 2
+ * node_demotion[1].preferred = 0
+ * node_demotion[2].preferred = <empty>
+ *
+ */
+static struct demotion_nodes *node_demotion __read_mostly;
+
/*
* Keep it simple by having direct mapping between
* tier index and rank value.
@@ -124,6 +195,7 @@ int node_create_and_set_memory_tier(int node, int tier)
current_tier = __node_get_memory_tier(node);
if (!current_tier) {
ret = __node_create_and_set_memory_tier(node, tier);
+ establish_migration_targets();
goto out;
}
@@ -138,6 +210,7 @@ int node_create_and_set_memory_tier(int node, int tier)
node_set(node, current_tier->nodelist);
goto out;
}
+ establish_migration_targets();
out:
mutex_unlock(&memory_tier_lock);
@@ -145,6 +218,223 @@ int node_create_and_set_memory_tier(int node, int tier)
}
EXPORT_SYMBOL_GPL(node_create_and_set_memory_tier);
+static int __node_set_memory_tier(int node, int tier)
+{
+ int ret = 0;
+ struct memory_tier *memtier;
+
+ memtier = __get_memory_tier_from_id(tier);
+ if (!memtier) {
+ ret = -EINVAL;
+ goto out;
+ }
+ node_set(node, memtier->nodelist);
+out:
+ return ret;
+}
+
+int node_set_memory_tier(int node, int tier)
+{
+ struct memory_tier *memtier;
+ int ret = 0;
+
+ mutex_lock(&memory_tier_lock);
+ memtier = __node_get_memory_tier(node);
+ /*
+ * if node is already part of the tier proceed with the
+ * current tier value, because we might want to establish
+ * new migration paths now. The node might be added to a tier
+ * before it was made part of N_MEMORY, hence estabilish_migration_targets
+ * will have skipped this node.
+ */
+ if (!memtier)
+ ret = __node_set_memory_tier(node, tier);
+ establish_migration_targets();
+
+ mutex_unlock(&memory_tier_lock);
+
+ return ret;
+}
+
+/**
+ * next_demotion_node() - Get the next node in the demotion path
+ * @node: The starting node to lookup the next node
+ *
+ * Return: node id for next memory node in the demotion path hierarchy
+ * from @node; NUMA_NO_NODE if @node is terminal. This does not keep
+ * @node online or guarantee that it *continues* to be the next demotion
+ * target.
+ */
+int next_demotion_node(int node)
+{
+ struct demotion_nodes *nd;
+ int target;
+
+ if (!node_demotion)
+ return NUMA_NO_NODE;
+
+ nd = &node_demotion[node];
+
+ /*
+ * node_demotion[] is updated without excluding this
+ * function from running.
+ *
+ * Make sure to use RCU over entire code blocks if
+ * node_demotion[] reads need to be consistent.
+ */
+ rcu_read_lock();
+ /*
+ * If there are multiple target nodes, just select one
+ * target node randomly.
+ *
+ * In addition, we can also use round-robin to select
+ * target node, but we should introduce another variable
+ * for node_demotion[] to record last selected target node,
+ * that may cause cache ping-pong due to the changing of
+ * last target node. Or introducing per-cpu data to avoid
+ * caching issue, which seems more complicated. So selecting
+ * target node randomly seems better until now.
+ */
+ target = node_random(&nd->preferred);
+ rcu_read_unlock();
+
+ return target;
+}
+
+/* Disable reclaim-based migration. */
+static void __disable_all_migrate_targets(void)
+{
+ int node;
+
+ for_each_node_state(node, N_MEMORY)
+ node_demotion[node].preferred = NODE_MASK_NONE;
+}
+
+static void disable_all_migrate_targets(void)
+{
+ __disable_all_migrate_targets();
+
+ /*
+ * Ensure that the "disable" is visible across the system.
+ * Readers will see either a combination of before+disable
+ * state or disable+after. They will never see before and
+ * after state together.
+ */
+ synchronize_rcu();
+}
+
+/*
+ * Find an automatic demotion target for all memory
+ * nodes. Failing here is OK. It might just indicate
+ * being at the end of a chain.
+ */
+static void establish_migration_targets(void)
+{
+ struct memory_tier *memtier;
+ struct demotion_nodes *nd;
+ int target = NUMA_NO_NODE, node;
+ int distance, best_distance;
+ nodemask_t used;
+
+ if (!node_demotion || !IS_ENABLED(CONFIG_MIGRATION))
+ return;
+
+ disable_all_migrate_targets();
+
+ for_each_node_state(node, N_MEMORY) {
+ best_distance = -1;
+ nd = &node_demotion[node];
+
+ memtier = __node_get_memory_tier(node);
+ if (!memtier || list_is_last(&memtier->list, &memory_tiers))
+ continue;
+ /*
+ * Get the next memtier to find the demotion node list.
+ */
+ memtier = list_next_entry(memtier, list);
+
+ /*
+ * find_next_best_node, use 'used' nodemask as a skip list.
+ * Add all memory nodes except the selected memory tier
+ * nodelist to skip list so that we find the best node from the
+ * memtier nodelist.
+ */
+ nodes_andnot(used, node_states[N_MEMORY], memtier->nodelist);
+
+ /*
+ * Find all the nodes in the memory tier node list of same best distance.
+ * add them to the preferred mask. We randomly select between nodes
+ * in the preferred mask when allocating pages during demotion.
+ */
+ do {
+ target = find_next_best_node(node, &used);
+ if (target == NUMA_NO_NODE)
+ break;
+
+ distance = node_distance(node, target);
+ if (distance == best_distance || best_distance == -1) {
+ best_distance = distance;
+ node_set(target, nd->preferred);
+ } else {
+ break;
+ }
+ } while (1);
+ }
+}
+
+/*
+ * This runs whether reclaim-based migration is enabled or not,
+ * which ensures that the user can turn reclaim-based migration
+ * at any time without needing to recalculate migration targets.
+ */
+static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
+ unsigned long action, void *_arg)
+{
+ struct memory_notify *arg = _arg;
+
+ /*
+ * Only update the node migration order when a node is
+ * changing status, like online->offline.
+ */
+ if (arg->status_change_nid < 0)
+ return notifier_from_errno(0);
+
+ switch (action) {
+ case MEM_OFFLINE:
+ /*
+ * In case we are moving out of N_MEMORY. Keep the node
+ * in the memory tier so that when we bring memory online,
+ * they appear in the right memory tier. We still need
+ * to rebuild the demotion order.
+ */
+ mutex_lock(&memory_tier_lock);
+ establish_migration_targets();
+ mutex_unlock(&memory_tier_lock);
+ break;
+ case MEM_ONLINE:
+ /*
+ * We ignore the error here, if the node already have the tier
+ * registered, we will continue to use that for the new memory
+ * we are adding here.
+ */
+ node_set_memory_tier(arg->status_change_nid, DEFAULT_MEMORY_TIER);
+ break;
+ }
+
+ return notifier_from_errno(0);
+}
+
+static void __init migrate_on_reclaim_init(void)
+{
+
+ if (IS_ENABLED(CONFIG_MIGRATION)) {
+ node_demotion = kcalloc(MAX_NUMNODES, sizeof(struct demotion_nodes),
+ GFP_KERNEL);
+ WARN_ON(!node_demotion);
+ }
+ hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
+}
+
static int __init memory_tier_init(void)
{
struct memory_tier *memtier;
@@ -162,6 +452,7 @@ static int __init memory_tier_init(void)
/* CPU only nodes are not part of memory tiers. */
memtier->nodelist = node_states[N_MEMORY];
+ migrate_on_reclaim_init();
return 0;
}
diff --git a/mm/migrate.c b/mm/migrate.c
index 29cacc217e38..0b554625a219 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2116,398 +2116,4 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
return 0;
}
#endif /* CONFIG_NUMA_BALANCING */
-
-/*
- * node_demotion[] example:
- *
- * Consider a system with two sockets. Each socket has
- * three classes of memory attached: fast, medium and slow.
- * Each memory class is placed in its own NUMA node. The
- * CPUs are placed in the node with the "fast" memory. The
- * 6 NUMA nodes (0-5) might be split among the sockets like
- * this:
- *
- * Socket A: 0, 1, 2
- * Socket B: 3, 4, 5
- *
- * When Node 0 fills up, its memory should be migrated to
- * Node 1. When Node 1 fills up, it should be migrated to
- * Node 2. The migration path start on the nodes with the
- * processors (since allocations default to this node) and
- * fast memory, progress through medium and end with the
- * slow memory:
- *
- * 0 -> 1 -> 2 -> stop
- * 3 -> 4 -> 5 -> stop
- *
- * This is represented in the node_demotion[] like this:
- *
- * { nr=1, nodes[0]=1 }, // Node 0 migrates to 1
- * { nr=1, nodes[0]=2 }, // Node 1 migrates to 2
- * { nr=0, nodes[0]=-1 }, // Node 2 does not migrate
- * { nr=1, nodes[0]=4 }, // Node 3 migrates to 4
- * { nr=1, nodes[0]=5 }, // Node 4 migrates to 5
- * { nr=0, nodes[0]=-1 }, // Node 5 does not migrate
- *
- * Moreover some systems may have multiple slow memory nodes.
- * Suppose a system has one socket with 3 memory nodes, node 0
- * is fast memory type, and node 1/2 both are slow memory
- * type, and the distance between fast memory node and slow
- * memory node is same. So the migration path should be:
- *
- * 0 -> 1/2 -> stop
- *
- * This is represented in the node_demotion[] like this:
- * { nr=2, {nodes[0]=1, nodes[1]=2} }, // Node 0 migrates to node 1 and node 2
- * { nr=0, nodes[0]=-1, }, // Node 1 dose not migrate
- * { nr=0, nodes[0]=-1, }, // Node 2 does not migrate
- */
-
-/*
- * Writes to this array occur without locking. Cycles are
- * not allowed: Node X demotes to Y which demotes to X...
- *
- * If multiple reads are performed, a single rcu_read_lock()
- * must be held over all reads to ensure that no cycles are
- * observed.
- */
-#define DEFAULT_DEMOTION_TARGET_NODES 15
-
-#if MAX_NUMNODES < DEFAULT_DEMOTION_TARGET_NODES
-#define DEMOTION_TARGET_NODES (MAX_NUMNODES - 1)
-#else
-#define DEMOTION_TARGET_NODES DEFAULT_DEMOTION_TARGET_NODES
-#endif
-
-struct demotion_nodes {
- unsigned short nr;
- short nodes[DEMOTION_TARGET_NODES];
-};
-
-static struct demotion_nodes *node_demotion __read_mostly;
-
-/**
- * next_demotion_node() - Get the next node in the demotion path
- * @node: The starting node to lookup the next node
- *
- * Return: node id for next memory node in the demotion path hierarchy
- * from @node; NUMA_NO_NODE if @node is terminal. This does not keep
- * @node online or guarantee that it *continues* to be the next demotion
- * target.
- */
-int next_demotion_node(int node)
-{
- struct demotion_nodes *nd;
- unsigned short target_nr, index;
- int target;
-
- if (!node_demotion)
- return NUMA_NO_NODE;
-
- nd = &node_demotion[node];
-
- /*
- * node_demotion[] is updated without excluding this
- * function from running. RCU doesn't provide any
- * compiler barriers, so the READ_ONCE() is required
- * to avoid compiler reordering or read merging.
- *
- * Make sure to use RCU over entire code blocks if
- * node_demotion[] reads need to be consistent.
- */
- rcu_read_lock();
- target_nr = READ_ONCE(nd->nr);
-
- switch (target_nr) {
- case 0:
- target = NUMA_NO_NODE;
- goto out;
- case 1:
- index = 0;
- break;
- default:
- /*
- * If there are multiple target nodes, just select one
- * target node randomly.
- *
- * In addition, we can also use round-robin to select
- * target node, but we should introduce another variable
- * for node_demotion[] to record last selected target node,
- * that may cause cache ping-pong due to the changing of
- * last target node. Or introducing per-cpu data to avoid
- * caching issue, which seems more complicated. So selecting
- * target node randomly seems better until now.
- */
- index = get_random_int() % target_nr;
- break;
- }
-
- target = READ_ONCE(nd->nodes[index]);
-
-out:
- rcu_read_unlock();
- return target;
-}
-
-/* Disable reclaim-based migration. */
-static void __disable_all_migrate_targets(void)
-{
- int node, i;
-
- if (!node_demotion)
- return;
-
- for_each_online_node(node) {
- node_demotion[node].nr = 0;
- for (i = 0; i < DEMOTION_TARGET_NODES; i++)
- node_demotion[node].nodes[i] = NUMA_NO_NODE;
- }
-}
-
-static void disable_all_migrate_targets(void)
-{
- __disable_all_migrate_targets();
-
- /*
- * Ensure that the "disable" is visible across the system.
- * Readers will see either a combination of before+disable
- * state or disable+after. They will never see before and
- * after state together.
- *
- * The before+after state together might have cycles and
- * could cause readers to do things like loop until this
- * function finishes. This ensures they can only see a
- * single "bad" read and would, for instance, only loop
- * once.
- */
- synchronize_rcu();
-}
-
-/*
- * Find an automatic demotion target for 'node'.
- * Failing here is OK. It might just indicate
- * being at the end of a chain.
- */
-static int establish_migrate_target(int node, nodemask_t *used,
- int best_distance)
-{
- int migration_target, index, val;
- struct demotion_nodes *nd;
-
- if (!node_demotion)
- return NUMA_NO_NODE;
-
- nd = &node_demotion[node];
-
- migration_target = find_next_best_node(node, used);
- if (migration_target == NUMA_NO_NODE)
- return NUMA_NO_NODE;
-
- /*
- * If the node has been set a migration target node before,
- * which means it's the best distance between them. Still
- * check if this node can be demoted to other target nodes
- * if they have a same best distance.
- */
- if (best_distance != -1) {
- val = node_distance(node, migration_target);
- if (val > best_distance)
- goto out_clear;
- }
-
- index = nd->nr;
- if (WARN_ONCE(index >= DEMOTION_TARGET_NODES,
- "Exceeds maximum demotion target nodes\n"))
- goto out_clear;
-
- nd->nodes[index] = migration_target;
- nd->nr++;
-
- return migration_target;
-out_clear:
- node_clear(migration_target, *used);
- return NUMA_NO_NODE;
-}
-
-/*
- * When memory fills up on a node, memory contents can be
- * automatically migrated to another node instead of
- * discarded at reclaim.
- *
- * Establish a "migration path" which will start at nodes
- * with CPUs and will follow the priorities used to build the
- * page allocator zonelists.
- *
- * The difference here is that cycles must be avoided. If
- * node0 migrates to node1, then neither node1, nor anything
- * node1 migrates to can migrate to node0. Also one node can
- * be migrated to multiple nodes if the target nodes all have
- * a same best-distance against the source node.
- *
- * This function can run simultaneously with readers of
- * node_demotion[]. However, it can not run simultaneously
- * with itself. Exclusion is provided by memory hotplug events
- * being single-threaded.
- */
-static void __set_migration_target_nodes(void)
-{
- nodemask_t next_pass;
- nodemask_t this_pass;
- nodemask_t used_targets = NODE_MASK_NONE;
- int node, best_distance;
-
- /*
- * Avoid any oddities like cycles that could occur
- * from changes in the topology. This will leave
- * a momentary gap when migration is disabled.
- */
- disable_all_migrate_targets();
-
- /*
- * Allocations go close to CPUs, first. Assume that
- * the migration path starts at the nodes with CPUs.
- */
- next_pass = node_states[N_CPU];
-again:
- this_pass = next_pass;
- next_pass = NODE_MASK_NONE;
- /*
- * To avoid cycles in the migration "graph", ensure
- * that migration sources are not future targets by
- * setting them in 'used_targets'. Do this only
- * once per pass so that multiple source nodes can
- * share a target node.
- *
- * 'used_targets' will become unavailable in future
- * passes. This limits some opportunities for
- * multiple source nodes to share a destination.
- */
- nodes_or(used_targets, used_targets, this_pass);
-
- for_each_node_mask(node, this_pass) {
- best_distance = -1;
-
- /*
- * Try to set up the migration path for the node, and the target
- * migration nodes can be multiple, so doing a loop to find all
- * the target nodes if they all have a best node distance.
- */
- do {
- int target_node =
- establish_migrate_target(node, &used_targets,
- best_distance);
-
- if (target_node == NUMA_NO_NODE)
- break;
-
- if (best_distance == -1)
- best_distance = node_distance(node, target_node);
-
- /*
- * Visit targets from this pass in the next pass.
- * Eventually, every node will have been part of
- * a pass, and will become set in 'used_targets'.
- */
- node_set(target_node, next_pass);
- } while (1);
- }
- /*
- * 'next_pass' contains nodes which became migration
- * targets in this pass. Make additional passes until
- * no more migrations targets are available.
- */
- if (!nodes_empty(next_pass))
- goto again;
-}
-
-/*
- * For callers that do not hold get_online_mems() already.
- */
-void set_migration_target_nodes(void)
-{
- get_online_mems();
- __set_migration_target_nodes();
- put_online_mems();
-}
-
-/*
- * This leaves migrate-on-reclaim transiently disabled between
- * the MEM_GOING_OFFLINE and MEM_OFFLINE events. This runs
- * whether reclaim-based migration is enabled or not, which
- * ensures that the user can turn reclaim-based migration at
- * any time without needing to recalculate migration targets.
- *
- * These callbacks already hold get_online_mems(). That is why
- * __set_migration_target_nodes() can be used as opposed to
- * set_migration_target_nodes().
- */
-#ifdef CONFIG_MEMORY_HOTPLUG
-static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
- unsigned long action, void *_arg)
-{
- struct memory_notify *arg = _arg;
-
- /*
- * Only update the node migration order when a node is
- * changing status, like online->offline. This avoids
- * the overhead of synchronize_rcu() in most cases.
- */
- if (arg->status_change_nid < 0)
- return notifier_from_errno(0);
-
- switch (action) {
- case MEM_GOING_OFFLINE:
- /*
- * Make sure there are not transient states where
- * an offline node is a migration target. This
- * will leave migration disabled until the offline
- * completes and the MEM_OFFLINE case below runs.
- */
- disable_all_migrate_targets();
- break;
- case MEM_OFFLINE:
- case MEM_ONLINE:
- /*
- * Recalculate the target nodes once the node
- * reaches its final state (online or offline).
- */
- __set_migration_target_nodes();
- break;
- case MEM_CANCEL_OFFLINE:
- /*
- * MEM_GOING_OFFLINE disabled all the migration
- * targets. Reenable them.
- */
- __set_migration_target_nodes();
- break;
- case MEM_GOING_ONLINE:
- case MEM_CANCEL_ONLINE:
- break;
- }
-
- return notifier_from_errno(0);
-}
-#endif
-
-void __init migrate_on_reclaim_init(void)
-{
- node_demotion = kcalloc(nr_node_ids,
- sizeof(struct demotion_nodes),
- GFP_KERNEL);
- WARN_ON(!node_demotion);
-#ifdef CONFIG_MEMORY_HOTPLUG
- hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
-#endif
- /*
- * At this point, all numa nodes with memory/CPus have their state
- * properly set, so we can build the demotion order now.
- * Let us hold the cpu_hotplug lock just, as we could possibily have
- * CPU hotplug events during boot.
- */
- cpus_read_lock();
- set_migration_target_nodes();
- cpus_read_unlock();
-}
#endif /* CONFIG_NUMA */
-
-
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 373d2730fcf2..35c6ff97cf29 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -28,7 +28,6 @@
#include <linux/mm_inline.h>
#include <linux/page_ext.h>
#include <linux/page_owner.h>
-#include <linux/migrate.h>
#include "internal.h"
@@ -2060,7 +2059,6 @@ static int vmstat_cpu_online(unsigned int cpu)
if (!node_state(cpu_to_node(cpu), N_CPU)) {
node_set_state(cpu_to_node(cpu), N_CPU);
- set_migration_target_nodes();
}
return 0;
@@ -2085,7 +2083,6 @@ static int vmstat_cpu_dead(unsigned int cpu)
return 0;
node_clear_state(node, N_CPU);
- set_migration_target_nodes();
return 0;
}
@@ -2118,7 +2115,6 @@ void __init init_mm_internals(void)
start_shepherd_timer();
#endif
- migrate_on_reclaim_init();
#ifdef CONFIG_PROC_FS
proc_create_seq("buddyinfo", 0444, NULL, &fragmentation_op);
proc_create_seq("pagetypeinfo", 0400, NULL, &pagetypeinfo_op);
--
2.36.1
With CONFIG_MIGRATION disabled return EINVAL on write.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
mm/memory-tiers.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 9c6b40d7e0bf..c3123a457d90 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
{
ssize_t ret;
+ if (!IS_ENABLED(CONFIG_MIGRATION))
+ return -EINVAL;
+
ret = kstrtobool(buf, &numa_demotion_enabled);
if (ret)
return ret;
--
2.36.1
This patch adds support to create memory tiers with specific rank
value from userspace. To avoid race while creating memory tiers
/sys/devices/system/memtier/create_tier_from_rank file is provided.
Writing to this file with a specific rank value creates a new memory
tier with the specified rank value.
Memory tiers created from userspace gets destroyed when the memory
tier nodelist becomes empty.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/memory-tiers.h | 3 +-
mm/memory-tiers.c | 74 ++++++++++++++++++++++++++++++++++++
2 files changed, 76 insertions(+), 1 deletion(-)
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index e70f0040d845..52896f5970b7 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -15,7 +15,8 @@
#define MEMORY_RANK_PMEM 100
#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM
-#define MAX_MEMORY_TIERS 3
+#define MAX_STATIC_MEMORY_TIERS 3
+#define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2)
extern bool numa_demotion_enabled;
int node_create_and_set_memory_tier(int node, int tier);
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 7bfdfac4d43e..de3b7403ae6f 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -6,6 +6,7 @@
#include <linux/memory-tiers.h>
#include <linux/random.h>
#include <linux/memory.h>
+#include <linux/idr.h>
#include "internal.h"
@@ -126,9 +127,12 @@ static const struct attribute_group *memory_tier_dev_groups[] = {
NULL
};
+static DEFINE_IDA(memtier_dev_id);
static void memory_tier_device_release(struct device *dev)
{
struct memory_tier *tier = to_memory_tier(dev);
+ if (tier->dev.id >= MAX_STATIC_MEMORY_TIERS)
+ ida_free(&memtier_dev_id, tier->dev.id);
kfree(tier);
}
@@ -195,6 +199,17 @@ static struct memory_tier *register_memory_tier(unsigned int tier,
return memtier;
}
+static void unregister_memory_tier(struct memory_tier *memtier)
+{
+ /*
+ * Don't destroy static memory tiers.
+ */
+ if (memtier->dev.id < MAX_STATIC_MEMORY_TIERS)
+ return;
+ list_del(&memtier->list);
+ device_unregister(&memtier->dev);
+}
+
static struct memory_tier *__node_get_memory_tier(int node)
{
struct memory_tier *memtier;
@@ -267,6 +282,10 @@ int node_create_and_set_memory_tier(int node, int tier)
node_set(node, current_tier->nodelist);
goto out;
}
+
+ if (nodes_empty(current_tier->nodelist))
+ unregister_memory_tier(current_tier);
+
establish_migration_targets();
out:
mutex_unlock(&memory_tier_lock);
@@ -350,6 +369,9 @@ int node_reset_memory_tier(int node, int tier)
goto out;
}
+ if (nodes_empty(current_tier->nodelist))
+ unregister_memory_tier(current_tier);
+
establish_migration_targets();
out:
mutex_unlock(&memory_tier_lock);
@@ -550,9 +572,61 @@ default_tier_show(struct device *dev, struct device_attribute *attr, char *buf)
}
static DEVICE_ATTR_RO(default_tier);
+static inline int memtier_alloc_id(void)
+{
+ return ida_alloc_range(&memtier_dev_id,
+ MAX_STATIC_MEMORY_TIERS,
+ MAX_MEMORY_TIERS - 1, GFP_KERNEL);
+}
+
+static ssize_t create_tier_from_rank_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret, tier, rank;
+ struct memory_tier *memtier;
+
+ ret = kstrtouint(buf, 10, &rank);
+ if (ret)
+ return ret;
+
+ if (rank == MEMORY_RANK_HBM_GPU ||
+ rank == MEMORY_RANK_DRAM ||
+ rank == MEMORY_RANK_PMEM)
+ return -EINVAL;
+
+ mutex_lock(&memory_tier_lock);
+ /*
+ * We don't support multiple tiers with same rank value
+ */
+ list_for_each_entry(memtier, &memory_tiers, list) {
+ if (memtier->rank == rank) {
+ ret = -EINVAL;
+ goto out;
+ }
+ }
+ tier = memtier_alloc_id();
+ if (tier < 0) {
+ ret = -ENOSPC;
+ goto out;
+ }
+ memtier = register_memory_tier(tier, rank);
+ if (IS_ERR(memtier)) {
+ ret = PTR_ERR(memtier);
+ goto out;
+ }
+
+ ret = count;
+out:
+ mutex_unlock(&memory_tier_lock);
+ return ret;
+}
+static DEVICE_ATTR_WO(create_tier_from_rank);
+
static struct attribute *memory_tier_attrs[] = {
&dev_attr_max_tier.attr,
&dev_attr_default_tier.attr,
+ &dev_attr_create_tier_from_rank.attr,
NULL
};
--
2.36.1
From: Jagdish Gediya <[email protected]>
Currently, a higher tier node can only be demoted to selected
nodes on the next lower tier as defined by the demotion path.
This strict, hard-coded demotion order does not work in all
use cases (e.g. some use cases may want to allow cross-socket
demotion to another node in the same demotion tier as a fallback
when the preferred demotion node is out of space). This demotion
order is also inconsistent with the page allocation fallback order
when all the nodes in a higher tier are out of space: The page
allocation can fall back to any node from any lower tier, whereas
the demotion order doesn't allow that currently.
This patch adds support to get all the allowed demotion targets
for a memory tier. demote_page_list() function is now modified
to utilize this allowed node mask as the fallback allocation mask.
Signed-off-by: Jagdish Gediya <[email protected]>
Signed-off-by: Aneesh Kumar K.V <[email protected]>
move allowed mask to memory tier
---
include/linux/memory-tiers.h | 9 ++++-
mm/memory-tiers.c | 75 +++++++++++++++++++++++++++++++++---
mm/vmscan.c | 56 ++++++++++++++++++++-------
3 files changed, 120 insertions(+), 20 deletions(-)
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 53f3e4c7cba8..47841379553c 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -3,11 +3,12 @@
#define _LINUX_MEMORY_TIERS_H
#include <linux/types.h>
+#include <linux/nodemask.h>
+#include <linux/mmzone.h>
#ifdef CONFIG_TIERED_MEMORY
#include <linux/device.h>
-#include <linux/nodemask.h>
#define MEMORY_TIER_HBM_GPU 0
#define MEMORY_TIER_DRAM 1
@@ -25,6 +26,7 @@ struct memory_tier {
struct list_head list;
struct device dev;
nodemask_t nodelist;
+ nodemask_t lower_tier_mask;
int rank;
};
@@ -36,6 +38,7 @@ int node_get_memory_tier_id(int node);
int node_reset_memory_tier(int node, int tier);
struct memory_tier *node_get_memory_tier(int node);
void node_put_memory_tier(struct memory_tier *memtier);
+void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
#else
@@ -45,6 +48,10 @@ static inline int next_demotion_node(int node)
return NUMA_NO_NODE;
}
+static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
+{
+ *targets = NODE_MASK_NONE;
+}
#endif /* CONFIG_TIERED_MEMORY */
#endif
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 429aa864edb0..b2ed16dcfb03 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -425,6 +425,24 @@ void node_put_memory_tier(struct memory_tier *memtier)
put_device(&memtier->dev);
}
+void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
+{
+ struct memory_tier *memtier;
+
+ /*
+ * pg_data_t.memtier updates includes a synchronize_rcu()
+ * which ensures that we either find NULL or a valid memtier
+ * in NODE_DATA. protect the access via rcu_read_lock();
+ */
+ rcu_read_lock();
+ memtier = rcu_dereference(pgdat->memtier);
+ if (memtier)
+ *targets = memtier->lower_tier_mask;
+ else
+ *targets = NODE_MASK_NONE;
+ rcu_read_unlock();
+}
+
/**
* next_demotion_node() - Get the next node in the demotion path
* @node: The starting node to lookup the next node
@@ -473,10 +491,18 @@ int next_demotion_node(int node)
/* Disable reclaim-based migration. */
static void __disable_all_migrate_targets(void)
{
+ pg_data_t *pgdat;
int node;
- for_each_node_state(node, N_MEMORY)
+ for_each_node_state(node, N_MEMORY) {
node_demotion[node].preferred = NODE_MASK_NONE;
+ /*
+ * We are holding memory_tier_lock, it is safe
+ * to access pgda->memtier.
+ */
+ pgdat = NODE_DATA(node);
+ pgdat->memtier->lower_tier_mask = NODE_MASK_NONE;
+ }
}
static void disable_all_migrate_targets(void)
@@ -503,10 +529,26 @@ static void establish_migration_targets(void)
struct demotion_nodes *nd;
int target = NUMA_NO_NODE, node;
int distance, best_distance;
- nodemask_t used;
-
- if (!node_demotion || !IS_ENABLED(CONFIG_MIGRATION))
- return;
+ nodemask_t used, lower_tier = NODE_MASK_NONE;
+
+ if (!node_demotion || !IS_ENABLED(CONFIG_MIGRATION)) {
+ pg_data_t *pgdat;
+
+ for_each_node_state(node, N_MEMORY) {
+ /*
+ * We are holding memory_tier_lock, it is safe
+ * to access pgda->memtier.
+ */
+ pgdat = NODE_DATA(node);
+ pgdat->memtier->lower_tier_mask = NODE_MASK_NONE;
+ }
+ /*
+ * Wait for read side to work with old values
+ * or see the updated NODE_MASK_NONE;
+ */
+ synchronize_rcu();
+ goto build_lower_tier_mask;
+ }
disable_all_migrate_targets();
@@ -549,6 +591,29 @@ static void establish_migration_targets(void)
}
} while (1);
}
+build_lower_tier_mask:
+ /*
+ * Now build the lower_tier mask for each node collecting node mask from
+ * all memory tier below it. This allows us to fallback demotion page
+ * allocation to a set of nodes that is closer the above selected
+ * perferred node.
+ */
+ list_for_each_entry(memtier, &memory_tiers, list)
+ nodes_or(lower_tier, lower_tier, memtier->nodelist);
+ /*
+ * Removes nodes not yet in N_MEMORY.
+ */
+ nodes_and(lower_tier, node_states[N_MEMORY], lower_tier);
+
+ list_for_each_entry(memtier, &memory_tiers, list) {
+ /*
+ * Keep removing current tier from lower_tier nodes,
+ * This will remove all nodes in current and above
+ * memory tier from the lower_tier mask.
+ */
+ nodes_andnot(lower_tier, lower_tier, memtier->nodelist);
+ memtier->lower_tier_mask = lower_tier;
+ }
}
/*
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 3a8f78277f99..2b213248effa 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1460,19 +1460,32 @@ static void folio_check_dirty_writeback(struct folio *folio,
mapping->a_ops->is_dirty_writeback(folio, dirty, writeback);
}
-static struct page *alloc_demote_page(struct page *page, unsigned long node)
+static struct page *alloc_demote_page(struct page *page, unsigned long private)
{
- struct migration_target_control mtc = {
- /*
- * Allocate from 'node', or fail quickly and quietly.
- * When this happens, 'page' will likely just be discarded
- * instead of migrated.
- */
- .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) |
- __GFP_THISNODE | __GFP_NOWARN |
- __GFP_NOMEMALLOC | GFP_NOWAIT,
- .nid = node
- };
+ struct page *target_page;
+ nodemask_t *allowed_mask;
+ struct migration_target_control *mtc;
+
+ mtc = (struct migration_target_control *)private;
+
+ allowed_mask = mtc->nmask;
+ /*
+ * make sure we allocate from the target node first also trying to
+ * reclaim pages from the target node via kswapd if we are low on
+ * free memory on target node. If we don't do this and if we have low
+ * free memory on the target memtier, we would start allocating pages
+ * from higher memory tiers without even forcing a demotion of cold
+ * pages from the target memtier. This can result in the kernel placing
+ * hotpages in higher memory tiers.
+ */
+ mtc->nmask = NULL;
+ mtc->gfp_mask |= __GFP_THISNODE;
+ target_page = alloc_migration_target(page, (unsigned long)&mtc);
+ if (target_page)
+ return target_page;
+
+ mtc->gfp_mask &= ~__GFP_THISNODE;
+ mtc->nmask = allowed_mask;
return alloc_migration_target(page, (unsigned long)&mtc);
}
@@ -1487,6 +1500,19 @@ static unsigned int demote_page_list(struct list_head *demote_pages,
{
int target_nid = next_demotion_node(pgdat->node_id);
unsigned int nr_succeeded;
+ nodemask_t allowed_mask;
+
+ struct migration_target_control mtc = {
+ /*
+ * Allocate from 'node', or fail quickly and quietly.
+ * When this happens, 'page' will likely just be discarded
+ * instead of migrated.
+ */
+ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOWARN |
+ __GFP_NOMEMALLOC | GFP_NOWAIT,
+ .nid = target_nid,
+ .nmask = &allowed_mask
+ };
if (list_empty(demote_pages))
return 0;
@@ -1494,10 +1520,12 @@ static unsigned int demote_page_list(struct list_head *demote_pages,
if (target_nid == NUMA_NO_NODE)
return 0;
+ node_get_allowed_targets(pgdat, &allowed_mask);
+
/* Demotion ignores all cpuset and mempolicy settings */
migrate_pages(demote_pages, alloc_demote_page, NULL,
- target_nid, MIGRATE_ASYNC, MR_DEMOTION,
- &nr_succeeded);
+ (unsigned long)&mtc, MIGRATE_ASYNC, MR_DEMOTION,
+ &nr_succeeded);
if (current_is_kswapd())
__count_vm_events(PGDEMOTE_KSWAPD, nr_succeeded);
--
2.36.1
With memory tiers support we can have memory only NUMA nodes
in the top tier from which we want to avoid promotion tracking NUMA
faults. Update node_is_toptier to work with memory tiers.
All NUMA nodes are by default top tier nodes. With lower memory
tiers added we consider all memory tiers above a memory tier having
CPU NUMA nodes as top memory tier
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/memory-tiers.h | 6 +++++
include/linux/node.h | 5 ----
mm/huge_memory.c | 1 +
mm/memory-tiers.c | 44 ++++++++++++++++++++++++++++++++++--
mm/migrate.c | 1 +
mm/mprotect.c | 1 +
6 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 47841379553c..de4098f6d5d5 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -39,6 +39,7 @@ int node_reset_memory_tier(int node, int tier);
struct memory_tier *node_get_memory_tier(int node);
void node_put_memory_tier(struct memory_tier *memtier);
void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
+bool node_is_toptier(int node);
#else
@@ -52,6 +53,11 @@ static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *target
{
*targets = NODE_MASK_NONE;
}
+
+static inline bool node_is_toptier(int node)
+{
+ return true;
+}
#endif /* CONFIG_TIERED_MEMORY */
#endif
diff --git a/include/linux/node.h b/include/linux/node.h
index 40d641a8bfb0..9ec680dd607f 100644
--- a/include/linux/node.h
+++ b/include/linux/node.h
@@ -185,9 +185,4 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,
#define to_node(device) container_of(device, struct node, dev)
-static inline bool node_is_toptier(int node)
-{
- return node_state(node, N_CPU);
-}
-
#endif /* _LINUX_NODE_H_ */
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index a77c78a2b6b5..294873d4be2b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -35,6 +35,7 @@
#include <linux/numa.h>
#include <linux/page_owner.h>
#include <linux/sched/sysctl.h>
+#include <linux/memory-tiers.h>
#include <asm/tlb.h>
#include <asm/pgalloc.h>
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index b2ed16dcfb03..0dae3114e22c 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -17,7 +17,7 @@ struct demotion_nodes {
static void establish_migration_targets(void);
static DEFINE_MUTEX(memory_tier_lock);
static LIST_HEAD(memory_tiers);
-
+static int top_tier_rank;
/*
* node_demotion[] examples:
*
@@ -126,7 +126,7 @@ static void memory_tier_device_release(struct device *dev)
if (tier->dev.id >= MAX_STATIC_MEMORY_TIERS)
ida_free(&memtier_dev_id, tier->dev.id);
- kfree(tier);
+ kfree_rcu(tier);
}
/*
@@ -443,6 +443,31 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
rcu_read_unlock();
}
+bool node_is_toptier(int node)
+{
+ bool toptier;
+ pg_data_t *pgdat;
+ struct memory_tier *memtier;
+
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return false;
+
+ rcu_read_lock();
+ memtier = rcu_dereference(pgdat->memtier);
+ if (!memtier) {
+ toptier = true;
+ goto out;
+ }
+ if (memtier->rank >= top_tier_rank)
+ toptier = true;
+ else
+ toptier = false;
+out:
+ rcu_read_unlock();
+ return toptier;
+}
+
/**
* next_demotion_node() - Get the next node in the demotion path
* @node: The starting node to lookup the next node
@@ -592,6 +617,21 @@ static void establish_migration_targets(void)
} while (1);
}
build_lower_tier_mask:
+ /*
+ * Promotion is allowed from a memory tier to higher
+ * memory tier only if the memory tier doesn't include
+ * compute. We want to skip promotion from a memory tier,
+ * if any node that is part of the memory tier have CPUs.
+ * Once we detect such a memory tier, we consider that tier
+ * as top tiper from which promotion is not allowed.
+ */
+ list_for_each_entry_reverse(memtier, &memory_tiers, list) {
+ nodes_and(used, node_states[N_CPU], memtier->nodelist);
+ if (!nodes_empty(used)) {
+ top_tier_rank = memtier->rank;
+ break;
+ }
+ }
/*
* Now build the lower_tier mask for each node collecting node mask from
* all memory tier below it. This allows us to fallback demotion page
diff --git a/mm/migrate.c b/mm/migrate.c
index 0b554625a219..78615c48fc0f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -50,6 +50,7 @@
#include <linux/memory.h>
#include <linux/random.h>
#include <linux/sched/sysctl.h>
+#include <linux/memory-tiers.h>
#include <asm/tlbflush.h>
diff --git a/mm/mprotect.c b/mm/mprotect.c
index ba5592655ee3..92a2fc0fa88b 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -31,6 +31,7 @@
#include <linux/pgtable.h>
#include <linux/sched/sysctl.h>
#include <linux/userfaultfd_k.h>
+#include <linux/memory-tiers.h>
#include <asm/cacheflush.h>
#include <asm/mmu_context.h>
#include <asm/tlbflush.h>
--
2.36.1
Add sysfs ABI documentation.
Signed-off-by: Wei Xu <[email protected]>
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
.../ABI/testing/sysfs-kernel-mm-memory-tiers | 87 +++++++++++++++++++
1 file changed, 87 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers b/Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
new file mode 100644
index 000000000000..b41d2977b0a5
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
@@ -0,0 +1,87 @@
+What: /sys/devices/system/memtier/
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Interface for tiered memory
+
+ This is the directory containing the information about memory tiers.
+
+ Each memory tier has its own subdirectory.
+
+ The order of memory tiers is determined by their rank values, not by
+ their memtier device names. A higher rank value means a higher tier.
+
+What: /sys/devices/system/memtier/default_tier
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Default memory tier
+
+ The default memory tier to which memory would get added via hotplug
+ if the NUMA node is not part of any memory tier
+
+What: /sys/devices/system/memtier/max_tier
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Maximum number of memory tiers supported
+
+ The max memory tier device ID we can create. Users can create memory
+ tiers in range [0 - max_tier)
+
+What: /sys/devices/system/memtier/create_tier_from_rank
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Interface to create memory tiers from userspace
+
+ Writing to this file with a rank value results in creation of
+ a new memory tier with the specified rank value. This is used
+ by userspace to create new memory tiers.
+
+What: /sys/devices/system/memtier/memtierN/
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Directory with details of a specific memory tier
+
+ This is the directory containing the information about a particular
+ memory tier, memtierN, where N is the memtier device ID (e.g. 0, 1).
+
+ The memtier device ID number itself is just an identifier and has no
+ special meaning, i.e. memtier device ID numbers do not determine the
+ order of memory tiers.
+
+What: /sys/devices/system/memtier/memtierN/rank
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Memory tier rank value
+
+
+ When read, list the "rank" value associated with memtierN.
+
+ "Rank" is an opaque value. Its absolute value doesn't have any
+ special meaning. But the rank values of different memtiers can be
+ compared with each other to determine the memory tier order.
+
+ For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and
+ their rank values are 100, 10, 50, then the memory tier order is:
+ memtier0 -> memtier2 -> memtier1, where memtier0 is the highest tier
+ and memtier1 is the lowest tier.
+
+ The rank value of each memtier should be unique.
+
+What: /sys/devices/system/memtier/memtierN/nodelist
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Memory tier nodelist
+
+
+ When read, list the memory nodes in the specified tier.
+
+What: /sys/devices/system/node/nodeN/memtier
+Date: June 2022
+Contact: Linux memory management mailing list <[email protected]>
+Description: Memory tier details for node N
+
+ When read, list the device ID of the memory tier that the node belongs
+ to. Its value is empty for a CPU-only NUMA node.
+
+ When written, the kernel moves the node into the specified memory
+ tier if the move is allowed. The tier assignments of all other
+ nodes are not affected.
--
2.36.1
From: Jagdish Gediya <[email protected]>
All N_MEMORY nodes are divided into 3 memoty tiers with rank value
MEMORY_RANK_HBM_GPU, MEMORY_RANK_DRAM and MEMORY_RANK_PMEM. By default,
All nodes are assigned to memory tier with rank value MEMORY_RANK_DRAM.
Demotion path for all N_MEMORY nodes is prepared based on the rank value
of memory tiers.
This patch adds documention for memory tiering introduction, its sysfs
interfaces and how demotion is performed based on memory tiers.
Suggested-by: Wei Xu <[email protected]>
Signed-off-by: Jagdish Gediya <[email protected]>
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
Documentation/admin-guide/mm/index.rst | 1 +
.../admin-guide/mm/memory-tiering.rst | 181 ++++++++++++++++++
2 files changed, 182 insertions(+)
create mode 100644 Documentation/admin-guide/mm/memory-tiering.rst
diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst
index c21b5823f126..3f211cbca8c3 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -32,6 +32,7 @@ the Linux memory management.
idle_page_tracking
ksm
memory-hotplug
+ memory-tiering
nommu-mmap
numa_memory_policy
numaperf
diff --git a/Documentation/admin-guide/mm/memory-tiering.rst b/Documentation/admin-guide/mm/memory-tiering.rst
new file mode 100644
index 000000000000..c2c7e95c1098
--- /dev/null
+++ b/Documentation/admin-guide/mm/memory-tiering.rst
@@ -0,0 +1,181 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _admin_guide_memory_tiering:
+
+============
+Memory tiers
+============
+
+This document describes explicit memory tiering support along with
+demotion based on memory tiers.
+
+Introduction
+============
+
+Many systems have multiple type of memory devices e.g. GPU, DRAM and
+PMEM. The memory subsystem of these systems can be called memory
+tiering system because the performance of the different types of
+memory is different. Memory tiers are defined based on hardware
+capabilities of memory nodes. Each memory tier is assigned a rank
+value that determines the memory tier position in demotion order.
+
+The memory tier assignment of each node is independent from each
+other. Moving a node from one tier to another tier doesn't affect
+the tier assignment of any other node.
+
+Memory tiers are used to build the demotion targets for nodes, a node
+can demote its pages to any node of any lower tiers.
+
+Memory tier rank
+=================
+
+Memory nodes are divided into below 3 types of memory tiers with rank value
+as shown base on their hardware characteristics.
+
+MEMORY_RANK_HBM_GPU
+MEMORY_RANK_DRAM
+MEMORY_RANK_PMEM
+
+Memory tiers initialization and (re)assignments
+===============================================
+
+By default, all nodes are assigned to memory tier with default rank
+DEFAULT_MEMORY_RANK which is 1 (MEMORY_RANK_DRAM). Memory tier of
+memory node can be either modified through sysfs or from driver. On
+hotplug, memory tier with default rank is assigned to memory node.
+
+Sysfs interfaces
+================
+
+Nodes belonging to specific tier can be read from,
+/sys/devices/system/memtier/memtierN/nodelist (Read-Only)
+
+Where N is 0 - 2.
+
+Example 1:
+For a system where Node 0 is CPU + DRAM nodes, Node 1 is HBM node,
+node 2 is PMEM node an ideal tier layout will be
+
+$ cat /sys/devices/system/memtier/memtier0/nodelist
+1
+$ cat /sys/devices/system/memtier/memtier1/nodelist
+0
+$ cat /sys/devices/system/memtier/memtier2/nodelist
+2
+
+Example 2:
+For a system where Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM
+nodes.
+
+$ cat /sys/devices/system/memtier/memtier0/nodelist
+cat: /sys/devices/system/memtier/memtier0/nodelist: No such file or
+directory
+$ cat /sys/devices/system/memtier/memtier1/nodelist
+0-1
+$ cat /sys/devices/system/memtier/memtier2/nodelist
+2-3
+
+Default memory tier can be read from,
+/sys/devices/system/memtier/default_tier (Read-Only)
+
+e.g.
+$ cat /sys/devices/system/memtier/default_tier
+memtier1
+
+Max memory tier can be read from,
+/sys/devices/system/memtier/max_tier (Read-Only)
+
+e.g.
+$ cat /sys/devices/system/memtier/max_tier
+3
+
+Individual node's memory tier can be read of set using,
+/sys/devices/system/node/nodeN/memtier (Read-Write)
+
+where N = node id
+
+When this interface is written, Node is moved from old memory tier
+to new memory tier and demotion targets for all N_MEMORY nodes are
+built again.
+
+For example 1 mentioned above,
+$ cat /sys/devices/system/node/node0/memtier
+1
+$ cat /sys/devices/system/node/node1/memtier
+0
+$ cat /sys/devices/system/node/node2/memtier
+2
+
+Creation of memory tiers from userspace
+/sys/devices/system/memtier/create_tier_from_rank (Read-write)
+
+Additional memory tiers can be created by writing a rank value to this file.
+This results in a new memory tier creation with specified rank value and empty nodelist.
+
+Demotion
+========
+
+In a system with DRAM and persistent memory, once DRAM
+fills up, reclaim will start and some of the DRAM contents will be
+thrown out even if there is a space in persistent memory.
+Consequently allocations will, at some point, start falling over to the slower
+persistent memory.
+
+That has two nasty properties. First, the newer allocations can end up in
+the slower persistent memory. Second, reclaimed data in DRAM are just
+discarded even if there are gobs of space in persistent memory that could
+be used.
+
+Instead of page being discarded during reclaim, it can be moved to
+persistent memory. Allowing page migration during reclaim enables
+these systems to migrate pages from fast(higher) tiers to slow(lower)
+tiers when the fast(higher) tier is under pressure.
+
+
+Enable/Disable demotion
+-----------------------
+
+By default demotion is disabled, it can be enabled/disabled using
+below sysfs interface,
+
+$ echo 0/1 or false/true > /sys/kernel/mm/numa/demotion_enabled
+
+preferred and allowed demotion nodes
+------------------------------------
+
+Preffered nodes for a specific N_MEMORY nodes are best nodes
+from next possible lower memory tier. Allowed nodes for any
+node are all the node available in all possible lower memory
+tiers.
+
+Example:
+
+For a system where Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM
+nodes,
+
+node distances:
+node 0 1 2 3
+ 0 10 20 30 40
+ 1 20 10 40 30
+ 2 30 40 10 40
+ 3 40 30 40 10
+
+memory_tiers[0] = <empty>
+memory_tiers[1] = 0-1
+memory_tiers[2] = 2-3
+
+node_demotion[0].preferred = 2
+node_demotion[0].allowed = 2, 3
+node_demotion[1].preferred = 3
+node_demotion[1].allowed = 3, 2
+node_demotion[2].preferred = <empty>
+node_demotion[2].allowed = <empty>
+node_demotion[3].preferred = <empty>
+node_demotion[3].allowed = <empty>
+
+Memory allocation for demotion
+------------------------------
+
+If page needs to be demoted from any node, the kernel 1st tries
+to allocate new page from node's preferred node and fallbacks to
+node's allowed targets in allocation fallback order.
--
2.36.1
On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> With CONFIG_MIGRATION disabled return EINVAL on write.
>
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> mm/memory-tiers.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 9c6b40d7e0bf..c3123a457d90 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
> {
> ssize_t ret;
>
>
> + if (!IS_ENABLED(CONFIG_MIGRATION))
> + return -EINVAL;
> +
How about enclose numa_demotion_enabled_xxx related code with CONFIG_MIGRATION?
> ret = kstrtobool(buf, &numa_demotion_enabled);
> if (ret)
> return ret;
Best Regards,
Huang, Ying
On 6/13/22 8:52 AM, Ying Huang wrote:
> Hi, Aneesh,
>
> On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
>> In the current kernel, memory tiers are defined implicitly via a
>> demotion path relationship between NUMA nodes, which is created
>> during the kernel initialization and updated when a NUMA node is
>> hot-added or hot-removed. The current implementation puts all
>> nodes with CPU into the top tier, and builds the tier hierarchy
>> tier-by-tier by establishing the per-node demotion targets based
>> on the distances between nodes.
>>
>> This current memory tier kernel interface needs to be improved for
>> several important use cases,
>>
>> The current tier initialization code always initializes
>> each memory-only NUMA node into a lower tier. But a memory-only
>> NUMA node may have a high performance memory device (e.g. a DRAM
>> device attached via CXL.mem or a DRAM-backed memory-only node on
>> a virtual machine) and should be put into a higher tier.
>>
>> The current tier hierarchy always puts CPU nodes into the top
>> tier. But on a system with HBM or GPU devices, the
>> memory-only NUMA nodes mapping these devices should be in the
>> top tier, and DRAM nodes with CPUs are better to be placed into the
>> next lower tier.
>>
>> With current kernel higher tier node can only be demoted to selected nodes on the
>> next lower tier as defined by the demotion path, not any other
>> node from any lower tier. This strict, hard-coded demotion order
>> does not work in all use cases (e.g. some use cases may want to
>> allow cross-socket demotion to another node in the same demotion
>> tier as a fallback when the preferred demotion node is out of
>> space), This demotion order is also inconsistent with the page
>> allocation fallback order when all the nodes in a higher tier are
>> out of space: The page allocation can fall back to any node from
>> any lower tier, whereas the demotion order doesn't allow that.
>>
>> The current kernel also don't provide any interfaces for the
>> userspace to learn about the memory tier hierarchy in order to
>> optimize its memory allocations.
>>
>> This patch series address the above by defining memory tiers explicitly.
>>
>> This patch introduce explicity memory tiers with ranks. The rank
>> value of a memory tier is used to derive the demotion order between
>> NUMA nodes. The memory tiers present in a system can be found at
>>
>> "Rank" is an opaque value. Its absolute value doesn't have any
>> special meaning. But the rank values of different memtiers can be
>> compared with each other to determine the memory tier order.
>>
>> For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and
>> their rank values are 300, 200, 100, then the memory tier order is:
>> memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier
>> and memtier2 is the lowest tier.
>>
>> The rank value of each memtier should be unique.
>>
>> A higher rank memory tier will appear first in the demotion order
>> than a lower rank memory tier. ie. while reclaim we choose a node
>> in higher rank memory tier to demote pages to as compared to a node
>> in a lower rank memory tier.
>>
>> This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2)
>> which are created by different kernel subsystems. The default memory
>> tier created by the kernel is memtier1. Once created these memory tiers
>> are not destroyed even if they don't have any NUMA nodes assigned to
>> them.
>>
>> This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
>>
>> [1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
>>
>> /sys/devices/system/memtier/memtierN/
>>
>> The nodes which are part of a specific memory tier can be listed
>> via
>> /sys/devices/system/memtier/memtierN/nodelist
>>
>> Suggested-by: Wei Xu <[email protected]>
>> Signed-off-by: Jagdish Gediya <[email protected]>
>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>> ---
>> include/linux/memory-tiers.h | 20 ++++++++
>> mm/Kconfig | 3 ++
>> mm/Makefile | 1 +
>> mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++
>> 4 files changed, 113 insertions(+)
>> create mode 100644 include/linux/memory-tiers.h
>> create mode 100644 mm/memory-tiers.c
>>
>> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
>> new file mode 100644
>> index 000000000000..e17f6b4ee177
>> --- /dev/null
>> +++ b/include/linux/memory-tiers.h
>> @@ -0,0 +1,20 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _LINUX_MEMORY_TIERS_H
>> +#define _LINUX_MEMORY_TIERS_H
>> +
>> +#ifdef CONFIG_TIERED_MEMORY
>> +
>> +#define MEMORY_TIER_HBM_GPU 0
>> +#define MEMORY_TIER_DRAM 1
>> +#define MEMORY_TIER_PMEM 2
>> +
>> +#define MEMORY_RANK_HBM_GPU 300
>> +#define MEMORY_RANK_DRAM 200
>> +#define MEMORY_RANK_PMEM 100
>> +
>> +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM
>> +#define MAX_MEMORY_TIERS 3
>> +
>> +#endif /* CONFIG_TIERED_MEMORY */
>> +
>> +#endif
>> diff --git a/mm/Kconfig b/mm/Kconfig
>> index 169e64192e48..bb5aa585ab41 100644
>> --- a/mm/Kconfig
>> +++ b/mm/Kconfig
>> @@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
>> config ARCH_ENABLE_THP_MIGRATION
>> bool
>>
>>
>> +config TIERED_MEMORY
>> + def_bool NUMA
>> +
>
> As Yang pointed out, why not just use CONFIG_NUMA? I suspect the
> added value of CONIFIG_TIRED_MEMORY.
>
I decided to use TIERED_MEMORY to bring more clarity. It should be same
now that we have moved CONFIG_MIGRATION dependencies to runtime. IMHO
having CONFIG_TIERED_MEMORY is better than using CONFIG_NUMA.
>> config HUGETLB_PAGE_SIZE_VARIABLE
>> def_bool n
>> help
>> diff --git a/mm/Makefile b/mm/Makefile
>> index 6f9ffa968a1a..482557fbc9d1 100644
>> --- a/mm/Makefile
>> +++ b/mm/Makefile
>> @@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
>> obj-$(CONFIG_FAILSLAB) += failslab.o
>> obj-$(CONFIG_MEMTEST) += memtest.o
>> obj-$(CONFIG_MIGRATION) += migrate.o
>> +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o
>> obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
>> obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
>> obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>> new file mode 100644
>> index 000000000000..d9fa955f208e
>> --- /dev/null
>> +++ b/mm/memory-tiers.c
>> @@ -0,0 +1,89 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +#include <linux/types.h>
>> +#include <linux/nodemask.h>
>> +#include <linux/slab.h>
>> +#include <linux/memory-tiers.h>
>> +
>> +struct memory_tier {
>> + struct list_head list;
>> + nodemask_t nodelist;
>> + int id;
>> + int rank;
>> +};
>> +
>> +static DEFINE_MUTEX(memory_tier_lock);
>> +static LIST_HEAD(memory_tiers);
>> +
>> +/*
>> + * Keep it simple by having direct mapping between
>> + * tier index and rank value.
>> + */
>> +static inline int get_rank_from_tier(unsigned int tier)
>> +{
>> + switch (tier) {
>> + case MEMORY_TIER_HBM_GPU:
>> + return MEMORY_RANK_HBM_GPU;
>> + case MEMORY_TIER_DRAM:
>> + return MEMORY_RANK_DRAM;
>> + case MEMORY_TIER_PMEM:
>> + return MEMORY_RANK_PMEM;
>> + }
>> + return -1;
>> +}
>> +
>> +static void insert_memory_tier(struct memory_tier *memtier)
>> +{
>> + struct list_head *ent;
>> + struct memory_tier *tmp_memtier;
>> +
>> + list_for_each(ent, &memory_tiers) {
>> + tmp_memtier = list_entry(ent, struct memory_tier, list);
>
> list_for_each_entry() ?
>
ent variable is used below. Hence I won't be able to use
list_for_each_entry.
>> + if (tmp_memtier->rank < memtier->rank) {
>> + list_add_tail(&memtier->list, ent);
>
>> + return;
>> + }
>> + }
>> + list_add_tail(&memtier->list, &memory_tiers);
>> +}
>> +
>
> IMHO, the locking requirements are needed here as comments to avoid
> confusing.
>
All those functions are called with memory_tier_lock_held. Infact all
list operations requires that lock held. What details do you suggest we
document? I can add extra comment to the mutex itself? Adding locking
details to all the functions will be duplicating the same details at
multiple places?
>> +static struct memory_tier *register_memory_tier(unsigned int tier,
>> + unsigned int rank)
>> +{
>> + struct memory_tier *memtier;
>> +
>> + if (tier >= MAX_MEMORY_TIERS)
>> + return ERR_PTR(-EINVAL);
>> +
>> + memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
>> + if (!memtier)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + memtier->id = tier;
>> + memtier->rank = rank;
>> +
>> + insert_memory_tier(memtier);
>> +
>> + return memtier;
>> +}
>> +
>> +static int __init memory_tier_init(void)
>> +{
>> + struct memory_tier *memtier;
>> +
>> + /*
>> + * Register only default memory tier to hide all empty
>> + * memory tier from sysfs.
>> + */
>> + memtier = register_memory_tier(DEFAULT_MEMORY_TIER,
>> + get_rank_from_tier(DEFAULT_MEMORY_TIER));
>> +
>> + if (IS_ERR(memtier))
>> + panic("%s() failed to register memory tier: %ld\n",
>> + __func__, PTR_ERR(memtier));
>> +
>> + /* CPU only nodes are not part of memory tiers. */
>> + memtier->nodelist = node_states[N_MEMORY];
>> +
>> + return 0;
>> +}
>> +subsys_initcall(memory_tier_init);
>
-aneesh
Hi, Aneesh,
On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> In the current kernel, memory tiers are defined implicitly via a
> demotion path relationship between NUMA nodes, which is created
> during the kernel initialization and updated when a NUMA node is
> hot-added or hot-removed. The current implementation puts all
> nodes with CPU into the top tier, and builds the tier hierarchy
> tier-by-tier by establishing the per-node demotion targets based
> on the distances between nodes.
>
> This current memory tier kernel interface needs to be improved for
> several important use cases,
>
> The current tier initialization code always initializes
> each memory-only NUMA node into a lower tier. But a memory-only
> NUMA node may have a high performance memory device (e.g. a DRAM
> device attached via CXL.mem or a DRAM-backed memory-only node on
> a virtual machine) and should be put into a higher tier.
>
> The current tier hierarchy always puts CPU nodes into the top
> tier. But on a system with HBM or GPU devices, the
> memory-only NUMA nodes mapping these devices should be in the
> top tier, and DRAM nodes with CPUs are better to be placed into the
> next lower tier.
>
> With current kernel higher tier node can only be demoted to selected nodes on the
> next lower tier as defined by the demotion path, not any other
> node from any lower tier. This strict, hard-coded demotion order
> does not work in all use cases (e.g. some use cases may want to
> allow cross-socket demotion to another node in the same demotion
> tier as a fallback when the preferred demotion node is out of
> space), This demotion order is also inconsistent with the page
> allocation fallback order when all the nodes in a higher tier are
> out of space: The page allocation can fall back to any node from
> any lower tier, whereas the demotion order doesn't allow that.
>
> The current kernel also don't provide any interfaces for the
> userspace to learn about the memory tier hierarchy in order to
> optimize its memory allocations.
>
> This patch series address the above by defining memory tiers explicitly.
>
> This patch introduce explicity memory tiers with ranks. The rank
> value of a memory tier is used to derive the demotion order between
> NUMA nodes. The memory tiers present in a system can be found at
>
> "Rank" is an opaque value. Its absolute value doesn't have any
> special meaning. But the rank values of different memtiers can be
> compared with each other to determine the memory tier order.
>
> For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and
> their rank values are 300, 200, 100, then the memory tier order is:
> memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier
> and memtier2 is the lowest tier.
>
> The rank value of each memtier should be unique.
>
> A higher rank memory tier will appear first in the demotion order
> than a lower rank memory tier. ie. while reclaim we choose a node
> in higher rank memory tier to demote pages to as compared to a node
> in a lower rank memory tier.
>
> This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2)
> which are created by different kernel subsystems. The default memory
> tier created by the kernel is memtier1. Once created these memory tiers
> are not destroyed even if they don't have any NUMA nodes assigned to
> them.
>
> This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
>
> [1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
>
> /sys/devices/system/memtier/memtierN/
>
> The nodes which are part of a specific memory tier can be listed
> via
> /sys/devices/system/memtier/memtierN/nodelist
>
> Suggested-by: Wei Xu <[email protected]>
> Signed-off-by: Jagdish Gediya <[email protected]>
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> include/linux/memory-tiers.h | 20 ++++++++
> mm/Kconfig | 3 ++
> mm/Makefile | 1 +
> mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++
> 4 files changed, 113 insertions(+)
> create mode 100644 include/linux/memory-tiers.h
> create mode 100644 mm/memory-tiers.c
>
> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
> new file mode 100644
> index 000000000000..e17f6b4ee177
> --- /dev/null
> +++ b/include/linux/memory-tiers.h
> @@ -0,0 +1,20 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _LINUX_MEMORY_TIERS_H
> +#define _LINUX_MEMORY_TIERS_H
> +
> +#ifdef CONFIG_TIERED_MEMORY
> +
> +#define MEMORY_TIER_HBM_GPU 0
> +#define MEMORY_TIER_DRAM 1
> +#define MEMORY_TIER_PMEM 2
> +
> +#define MEMORY_RANK_HBM_GPU 300
> +#define MEMORY_RANK_DRAM 200
> +#define MEMORY_RANK_PMEM 100
> +
> +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM
> +#define MAX_MEMORY_TIERS 3
> +
> +#endif /* CONFIG_TIERED_MEMORY */
> +
> +#endif
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 169e64192e48..bb5aa585ab41 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
> config ARCH_ENABLE_THP_MIGRATION
> bool
>
>
> +config TIERED_MEMORY
> + def_bool NUMA
> +
As Yang pointed out, why not just use CONFIG_NUMA? I suspect the
added value of CONIFIG_TIRED_MEMORY.
> config HUGETLB_PAGE_SIZE_VARIABLE
> def_bool n
> help
> diff --git a/mm/Makefile b/mm/Makefile
> index 6f9ffa968a1a..482557fbc9d1 100644
> --- a/mm/Makefile
> +++ b/mm/Makefile
> @@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
> obj-$(CONFIG_FAILSLAB) += failslab.o
> obj-$(CONFIG_MEMTEST) += memtest.o
> obj-$(CONFIG_MIGRATION) += migrate.o
> +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o
> obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
> obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
> obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> new file mode 100644
> index 000000000000..d9fa955f208e
> --- /dev/null
> +++ b/mm/memory-tiers.c
> @@ -0,0 +1,89 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <linux/types.h>
> +#include <linux/nodemask.h>
> +#include <linux/slab.h>
> +#include <linux/memory-tiers.h>
> +
> +struct memory_tier {
> + struct list_head list;
> + nodemask_t nodelist;
> + int id;
> + int rank;
> +};
> +
> +static DEFINE_MUTEX(memory_tier_lock);
> +static LIST_HEAD(memory_tiers);
> +
> +/*
> + * Keep it simple by having direct mapping between
> + * tier index and rank value.
> + */
> +static inline int get_rank_from_tier(unsigned int tier)
> +{
> + switch (tier) {
> + case MEMORY_TIER_HBM_GPU:
> + return MEMORY_RANK_HBM_GPU;
> + case MEMORY_TIER_DRAM:
> + return MEMORY_RANK_DRAM;
> + case MEMORY_TIER_PMEM:
> + return MEMORY_RANK_PMEM;
> + }
> + return -1;
> +}
> +
> +static void insert_memory_tier(struct memory_tier *memtier)
> +{
> + struct list_head *ent;
> + struct memory_tier *tmp_memtier;
> +
> + list_for_each(ent, &memory_tiers) {
> + tmp_memtier = list_entry(ent, struct memory_tier, list);
list_for_each_entry() ?
> + if (tmp_memtier->rank < memtier->rank) {
> + list_add_tail(&memtier->list, ent);
> + return;
> + }
> + }
> + list_add_tail(&memtier->list, &memory_tiers);
> +}
> +
IMHO, the locking requirements are needed here as comments to avoid
confusing.
> +static struct memory_tier *register_memory_tier(unsigned int tier,
> + unsigned int rank)
> +{
> + struct memory_tier *memtier;
> +
> + if (tier >= MAX_MEMORY_TIERS)
> + return ERR_PTR(-EINVAL);
> +
> + memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
> + if (!memtier)
> + return ERR_PTR(-ENOMEM);
> +
> + memtier->id = tier;
> + memtier->rank = rank;
> +
> + insert_memory_tier(memtier);
> +
> + return memtier;
> +}
> +
> +static int __init memory_tier_init(void)
> +{
> + struct memory_tier *memtier;
> +
> + /*
> + * Register only default memory tier to hide all empty
> + * memory tier from sysfs.
> + */
> + memtier = register_memory_tier(DEFAULT_MEMORY_TIER,
> + get_rank_from_tier(DEFAULT_MEMORY_TIER));
> +
> + if (IS_ERR(memtier))
> + panic("%s() failed to register memory tier: %ld\n",
> + __func__, PTR_ERR(memtier));
> +
> + /* CPU only nodes are not part of memory tiers. */
> + memtier->nodelist = node_states[N_MEMORY];
> +
> + return 0;
> +}
> +subsys_initcall(memory_tier_init);
Best Regards,
Huang, Ying
On 6/13/22 8:56 AM, Ying Huang wrote:
> On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
>> With CONFIG_MIGRATION disabled return EINVAL on write.
>>
>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>> ---
>> mm/memory-tiers.c | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>> index 9c6b40d7e0bf..c3123a457d90 100644
>> --- a/mm/memory-tiers.c
>> +++ b/mm/memory-tiers.c
>> @@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
>> {
>> ssize_t ret;
>>
>>
>> + if (!IS_ENABLED(CONFIG_MIGRATION))
>> + return -EINVAL;
>> +
>
> How about enclose numa_demotion_enabled_xxx related code with CONFIG_MIGRATION?
>
IIUC there is a desire to use IS_ENABLED() in the kernel instead of
#ifdef since that helps in more compile time checks. Because there are
no dead codes during compile now with IS_ENABLED().
W.r.t leaving the sysfs file visible even when CONFIG_MIGRATION is
disabled, I was thinking it gives better visibility into numa_demotion
status. I could switch to hide numa_demotion file if that is desirable.
>> ret = kstrtobool(buf, &numa_demotion_enabled);
>> if (ret)
>> return ret;
>
-aneesh
On Mon, 2022-06-13 at 09:01 +0530, Aneesh Kumar K V wrote:
> On 6/13/22 8:52 AM, Ying Huang wrote:
> > Hi, Aneesh,
> >
> > On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> > > In the current kernel, memory tiers are defined implicitly via a
> > > demotion path relationship between NUMA nodes, which is created
> > > during the kernel initialization and updated when a NUMA node is
> > > hot-added or hot-removed. The current implementation puts all
> > > nodes with CPU into the top tier, and builds the tier hierarchy
> > > tier-by-tier by establishing the per-node demotion targets based
> > > on the distances between nodes.
> > >
> > > This current memory tier kernel interface needs to be improved for
> > > several important use cases,
> > >
> > > The current tier initialization code always initializes
> > > each memory-only NUMA node into a lower tier. But a memory-only
> > > NUMA node may have a high performance memory device (e.g. a DRAM
> > > device attached via CXL.mem or a DRAM-backed memory-only node on
> > > a virtual machine) and should be put into a higher tier.
> > >
> > > The current tier hierarchy always puts CPU nodes into the top
> > > tier. But on a system with HBM or GPU devices, the
> > > memory-only NUMA nodes mapping these devices should be in the
> > > top tier, and DRAM nodes with CPUs are better to be placed into the
> > > next lower tier.
> > >
> > > With current kernel higher tier node can only be demoted to selected nodes on the
> > > next lower tier as defined by the demotion path, not any other
> > > node from any lower tier. This strict, hard-coded demotion order
> > > does not work in all use cases (e.g. some use cases may want to
> > > allow cross-socket demotion to another node in the same demotion
> > > tier as a fallback when the preferred demotion node is out of
> > > space), This demotion order is also inconsistent with the page
> > > allocation fallback order when all the nodes in a higher tier are
> > > out of space: The page allocation can fall back to any node from
> > > any lower tier, whereas the demotion order doesn't allow that.
> > >
> > > The current kernel also don't provide any interfaces for the
> > > userspace to learn about the memory tier hierarchy in order to
> > > optimize its memory allocations.
> > >
> > > This patch series address the above by defining memory tiers explicitly.
> > >
> > > This patch introduce explicity memory tiers with ranks. The rank
> > > value of a memory tier is used to derive the demotion order between
> > > NUMA nodes. The memory tiers present in a system can be found at
> > >
> > > "Rank" is an opaque value. Its absolute value doesn't have any
> > > special meaning. But the rank values of different memtiers can be
> > > compared with each other to determine the memory tier order.
> > >
> > > For example, if we have 3 memtiers: memtier0, memtier1, memiter2, and
> > > their rank values are 300, 200, 100, then the memory tier order is:
> > > memtier0 -> memtier1 -> memtier2, where memtier0 is the highest tier
> > > and memtier2 is the lowest tier.
> > >
> > > The rank value of each memtier should be unique.
> > >
> > > A higher rank memory tier will appear first in the demotion order
> > > than a lower rank memory tier. ie. while reclaim we choose a node
> > > in higher rank memory tier to demote pages to as compared to a node
> > > in a lower rank memory tier.
> > >
> > > This patchset introduce 3 memory tiers (memtier0, memtier1 and memtier2)
> > > which are created by different kernel subsystems. The default memory
> > > tier created by the kernel is memtier1. Once created these memory tiers
> > > are not destroyed even if they don't have any NUMA nodes assigned to
> > > them.
> > >
> > > This patch is based on the proposal sent by Wei Xu <[email protected]> at [1].
> > >
> > > [1] https://lore.kernel.org/linux-mm/CAAPL-u9Wv+nH1VOZTj=9p9S70Y3Qz3+63EkqncRDdHfubsrjfw@mail.gmail.com
> > >
> > > /sys/devices/system/memtier/memtierN/
> > >
> > > The nodes which are part of a specific memory tier can be listed
> > > via
> > > /sys/devices/system/memtier/memtierN/nodelist
> > >
> > > Suggested-by: Wei Xu <[email protected]>
> > > Signed-off-by: Jagdish Gediya <[email protected]>
> > > Signed-off-by: Aneesh Kumar K.V <[email protected]>
> > > ---
> > > include/linux/memory-tiers.h | 20 ++++++++
> > > mm/Kconfig | 3 ++
> > > mm/Makefile | 1 +
> > > mm/memory-tiers.c | 89 ++++++++++++++++++++++++++++++++++++
> > > 4 files changed, 113 insertions(+)
> > > create mode 100644 include/linux/memory-tiers.h
> > > create mode 100644 mm/memory-tiers.c
> > >
> > > diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
> > > new file mode 100644
> > > index 000000000000..e17f6b4ee177
> > > --- /dev/null
> > > +++ b/include/linux/memory-tiers.h
> > > @@ -0,0 +1,20 @@
> > > +/* SPDX-License-Identifier: GPL-2.0 */
> > > +#ifndef _LINUX_MEMORY_TIERS_H
> > > +#define _LINUX_MEMORY_TIERS_H
> > > +
> > > +#ifdef CONFIG_TIERED_MEMORY
> > > +
> > > +#define MEMORY_TIER_HBM_GPU 0
> > > +#define MEMORY_TIER_DRAM 1
> > > +#define MEMORY_TIER_PMEM 2
> > > +
> > > +#define MEMORY_RANK_HBM_GPU 300
> > > +#define MEMORY_RANK_DRAM 200
> > > +#define MEMORY_RANK_PMEM 100
> > > +
> > > +#define DEFAULT_MEMORY_TIER MEMORY_TIER_DRAM
> > > +#define MAX_MEMORY_TIERS 3
> > > +
> > > +#endif /* CONFIG_TIERED_MEMORY */
> > > +
> > > +#endif
> > > diff --git a/mm/Kconfig b/mm/Kconfig
> > > index 169e64192e48..bb5aa585ab41 100644
> > > --- a/mm/Kconfig
> > > +++ b/mm/Kconfig
> > > @@ -614,6 +614,9 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
> > > config ARCH_ENABLE_THP_MIGRATION
> > > bool
> > >
> > >
> > >
> > > +config TIERED_MEMORY
> > > + def_bool NUMA
> > > +
> >
> > As Yang pointed out, why not just use CONFIG_NUMA? I suspect the
> > added value of CONIFIG_TIRED_MEMORY.
> >
>
> I decided to use TIERED_MEMORY to bring more clarity. It should be same
> now that we have moved CONFIG_MIGRATION dependencies to runtime. IMHO
> having CONFIG_TIERED_MEMORY is better than using CONFIG_NUMA.
I don't think CONFIG_TIERED_MEMORY bring no much value. It's better
to use CONFIG_NUMA directly. But this is just my opinion.
> > > config HUGETLB_PAGE_SIZE_VARIABLE
> > > def_bool n
> > > help
> > > diff --git a/mm/Makefile b/mm/Makefile
> > > index 6f9ffa968a1a..482557fbc9d1 100644
> > > --- a/mm/Makefile
> > > +++ b/mm/Makefile
> > > @@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
> > > obj-$(CONFIG_FAILSLAB) += failslab.o
> > > obj-$(CONFIG_MEMTEST) += memtest.o
> > > obj-$(CONFIG_MIGRATION) += migrate.o
> > > +obj-$(CONFIG_TIERED_MEMORY) += memory-tiers.o
> > > obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
> > > obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
> > > obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
> > > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> > > new file mode 100644
> > > index 000000000000..d9fa955f208e
> > > --- /dev/null
> > > +++ b/mm/memory-tiers.c
> > > @@ -0,0 +1,89 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +#include <linux/types.h>
> > > +#include <linux/nodemask.h>
> > > +#include <linux/slab.h>
> > > +#include <linux/memory-tiers.h>
> > > +
> > > +struct memory_tier {
> > > + struct list_head list;
> > > + nodemask_t nodelist;
> > > + int id;
> > > + int rank;
> > > +};
> > > +
> > > +static DEFINE_MUTEX(memory_tier_lock);
> > > +static LIST_HEAD(memory_tiers);
> > > +
> > > +/*
> > > + * Keep it simple by having direct mapping between
> > > + * tier index and rank value.
> > > + */
> > > +static inline int get_rank_from_tier(unsigned int tier)
> > > +{
> > > + switch (tier) {
> > > + case MEMORY_TIER_HBM_GPU:
> > > + return MEMORY_RANK_HBM_GPU;
> > > + case MEMORY_TIER_DRAM:
> > > + return MEMORY_RANK_DRAM;
> > > + case MEMORY_TIER_PMEM:
> > > + return MEMORY_RANK_PMEM;
> > > + }
> > > + return -1;
> > > +}
> > > +
> > > +static void insert_memory_tier(struct memory_tier *memtier)
> > > +{
> > > + struct list_head *ent;
> > > + struct memory_tier *tmp_memtier;
> > > +
> > > + list_for_each(ent, &memory_tiers) {
> > > + tmp_memtier = list_entry(ent, struct memory_tier, list);
> >
> > list_for_each_entry() ?
> >
>
> ent variable is used below. Hence I won't be able to use
> list_for_each_entry.
ent == &tmp_memtier->list ?
> > > + if (tmp_memtier->rank < memtier->rank) {
> > > + list_add_tail(&memtier->list, ent);
> >
> > > + return;
> > > + }
> > > + }
> > > + list_add_tail(&memtier->list, &memory_tiers);
> > > +}
> > > +
> >
> > IMHO, the locking requirements are needed here as comments to avoid
> > confusing.
> >
>
> All those functions are called with memory_tier_lock_held. Infact all
> list operations requires that lock held. What details do you suggest we
> document? I can add extra comment to the mutex itself? Adding locking
> details to all the functions will be duplicating the same details at
> multiple places?
memory_tier_lock isn't held to call register_memory_tier() in this
patch. That will cause confusion.
> > > +static struct memory_tier *register_memory_tier(unsigned int tier,
> > > + unsigned int rank)
> > > +{
> > > + struct memory_tier *memtier;
> > > +
> > > + if (tier >= MAX_MEMORY_TIERS)
> > > + return ERR_PTR(-EINVAL);
> > > +
> > > + memtier = kzalloc(sizeof(struct memory_tier), GFP_KERNEL);
> > > + if (!memtier)
> > > + return ERR_PTR(-ENOMEM);
> > > +
> > > + memtier->id = tier;
> > > + memtier->rank = rank;
> > > +
> > > + insert_memory_tier(memtier);
> > > +
> > > + return memtier;
> > > +}
> > > +
> > > +static int __init memory_tier_init(void)
> > > +{
> > > + struct memory_tier *memtier;
> > > +
> > > + /*
> > > + * Register only default memory tier to hide all empty
> > > + * memory tier from sysfs.
> > > + */
> > > + memtier = register_memory_tier(DEFAULT_MEMORY_TIER,
> > > + get_rank_from_tier(DEFAULT_MEMORY_TIER));
> > > +
> > > + if (IS_ERR(memtier))
> > > + panic("%s() failed to register memory tier: %ld\n",
> > > + __func__, PTR_ERR(memtier));
> > > +
> > > + /* CPU only nodes are not part of memory tiers. */
> > > + memtier->nodelist = node_states[N_MEMORY];
> > > +
> > > + return 0;
> > > +}
> > > +subsys_initcall(memory_tier_init);
> >
Best Regards,
Huang, Ying
On Mon, 2022-06-13 at 09:05 +0530, Aneesh Kumar K V wrote:
> On 6/13/22 8:56 AM, Ying Huang wrote:
> > On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> > > With CONFIG_MIGRATION disabled return EINVAL on write.
> > >
> > > Signed-off-by: Aneesh Kumar K.V <[email protected]>
> > > ---
> > > mm/memory-tiers.c | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> > > index 9c6b40d7e0bf..c3123a457d90 100644
> > > --- a/mm/memory-tiers.c
> > > +++ b/mm/memory-tiers.c
> > > @@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
> > > {
> > > ssize_t ret;
> > >
> > >
> > >
> > > + if (!IS_ENABLED(CONFIG_MIGRATION))
> > > + return -EINVAL;
> > > +
> >
> > How about enclose numa_demotion_enabled_xxx related code with CONFIG_MIGRATION?
> >
>
> IIUC there is a desire to use IS_ENABLED() in the kernel instead of
> #ifdef since that helps in more compile time checks. Because there are
> no dead codes during compile now with IS_ENABLED().
IS_ENABLED() is used to reduce usage of "#ifdef" in ".c" file,
especially inside a function. We have good build test coverage with
0Day now.
To avoid code size inflate, it's better to use #ifdef CONFIG_MIGRATION.
> W.r.t leaving the sysfs file visible even when CONFIG_MIGRATION is
> disabled, I was thinking it gives better visibility into numa_demotion
> status. I could switch to hide numa_demotion file if that is desirable.
>
> > > ret = kstrtobool(buf, &numa_demotion_enabled);
> > > if (ret)
> > > return ret;
> >
Best Regards,
Huang, Ying
On 6/13/22 11:03 AM, Ying Huang wrote:
> On Mon, 2022-06-13 at 09:05 +0530, Aneesh Kumar K V wrote:
>> On 6/13/22 8:56 AM, Ying Huang wrote:
>>> On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
>>>> With CONFIG_MIGRATION disabled return EINVAL on write.
>>>>
>>>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>>>> ---
>>>> mm/memory-tiers.c | 3 +++
>>>> 1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>>>> index 9c6b40d7e0bf..c3123a457d90 100644
>>>> --- a/mm/memory-tiers.c
>>>> +++ b/mm/memory-tiers.c
>>>> @@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
>>>> {
>>>> ssize_t ret;
>>>>
>>>>
>>>>
>>>> + if (!IS_ENABLED(CONFIG_MIGRATION))
>>>> + return -EINVAL;
>>>> +
>>>
>>> How about enclose numa_demotion_enabled_xxx related code with CONFIG_MIGRATION?
>>>
>>
>> IIUC there is a desire to use IS_ENABLED() in the kernel instead of
>> #ifdef since that helps in more compile time checks. Because there are
>> no dead codes during compile now with IS_ENABLED().
>
> IS_ENABLED() is used to reduce usage of "#ifdef" in ".c" file,
> especially inside a function. We have good build test coverage with
> 0Day now.
>
> To avoid code size inflate, it's better to use #ifdef CONFIG_MIGRATION.
>
For a diff like below I am finding IS_ENABLED better.
size memory-tiers.o.isenabled memory-tiers.o
text data bss dec hex filename
4776 989 5 5770 168a memory-tiers.o.isenabled
5257 990 5 6252 186c memory-tiers.o
modified mm/memory-tiers.c
@@ -710,12 +710,11 @@ static int __meminit
migrate_on_reclaim_callback(struct notifier_block *self,
static void __init migrate_on_reclaim_init(void)
{
-
- if (IS_ENABLED(CONFIG_MIGRATION)) {
+#ifdef CONFIG_MIGRATION
node_demotion = kcalloc(MAX_NUMNODES, sizeof(struct demotion_nodes),
GFP_KERNEL);
WARN_ON(!node_demotion);
- }
+#endif
hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
}
@@ -844,14 +843,19 @@ static ssize_t numa_demotion_enabled_show(struct
kobject *kobj,
numa_demotion_enabled ? "true" : "false");
}
+#ifdef CONFIG_MIGRATION
static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
struct kobj_attribute *attr,
const char *buf, size_t count)
{
- ssize_t ret;
-
- if (!IS_ENABLED(CONFIG_MIGRATION))
- return -EINVAL;
+ return -EINVAL;
+}
+#else
+static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ ssize_t ret;
ret = kstrtobool(buf, &numa_demotion_enabled);
if (ret)
@@ -859,6 +863,7 @@ static ssize_t numa_demotion_enabled_store(struct
kobject *kobj,
return count;
}
+#endif
static struct kobj_attribute numa_demotion_enabled_attr =
__ATTR(demotion_enabled, 0644, numa_demotion_enabled_show,
I also find that #ifdef config not easier to the eyes. If there is a
large code that we can end up #ifdef out, then it might be worth it.
IIUC, we might want to keep the establish_migration target to find
top_tier rank and lower_tier mask. Once we do that only thing that we
could comment out is the node_demotion sysfs creation and I was
considering to keep that even if migration is disabled with a write to
the file returning EINVAL. I could switch that if you strongly feel that
we should hide node_demotion sysfs file.
-aneesh
On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> Also update different helpes to use NODE_DATA()->memtier. Since
> node specific memtier can change based on the reassignment of
> NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
> needs to under an rcu read lock of memory_tier_lock.
>
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> include/linux/memory-tiers.h | 14 +++++
> include/linux/mmzone.h | 3 ++
> mm/memory-tiers.c | 102 ++++++++++++++++++++++++++---------
> 3 files changed, 94 insertions(+), 25 deletions(-)
>
> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
> index 52896f5970b7..53f3e4c7cba8 100644
> --- a/include/linux/memory-tiers.h
> +++ b/include/linux/memory-tiers.h
> @@ -6,6 +6,9 @@
>
>
> #ifdef CONFIG_TIERED_MEMORY
>
>
> +#include <linux/device.h>
> +#include <linux/nodemask.h>
> +
> #define MEMORY_TIER_HBM_GPU 0
> #define MEMORY_TIER_DRAM 1
> #define MEMORY_TIER_PMEM 2
> @@ -18,13 +21,24 @@
> #define MAX_STATIC_MEMORY_TIERS 3
> #define MAX_MEMORY_TIERS (MAX_STATIC_MEMORY_TIERS + 2)
>
>
> +struct memory_tier {
> + struct list_head list;
> + struct device dev;
> + nodemask_t nodelist;
> + int rank;
> +};
> +
I suggest to use two data structure,
struct memory_tier {
struct list_head list;
nodemask_t nodelist;
int rank;
};
struct memory_tier_dev {
struct list_head list;
struct device dev;
struct memory_tier *tier;
};
Then we can put struct memory_tier here and still hide struct
memory_tier_dev in memory_tiers.c. In this way, we don't need to
force all struct memory_tier users to compile the entire driver core
headers. And we can separate the user space interface implementation
from the other part of the kernel.
> extern bool numa_demotion_enabled;
> int node_create_and_set_memory_tier(int node, int tier);
> int next_demotion_node(int node);
> int node_set_memory_tier(int node, int tier);
> int node_get_memory_tier_id(int node);
> int node_reset_memory_tier(int node, int tier);
> +struct memory_tier *node_get_memory_tier(int node);
> +void node_put_memory_tier(struct memory_tier *memtier);
I don't find caller of these 2 functions in series. Can we remove
these functions?
Best Regards,
Huang, Ying
[snip]
On Mon, Jun 13, 2022 at 01:30:08PM +0800, Ying Huang wrote:
> On Mon, 2022-06-13 at 09:01 +0530, Aneesh Kumar K V wrote:
> > On 6/13/22 8:52 AM, Ying Huang wrote:
> > > On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> > > > +config TIERED_MEMORY
> > > > + def_bool NUMA
> > > > +
> > >
> > > As Yang pointed out, why not just use CONFIG_NUMA? I suspect the
> > > added value of CONIFIG_TIRED_MEMORY.
> >
> > I decided to use TIERED_MEMORY to bring more clarity. It should be same
> > now that we have moved CONFIG_MIGRATION dependencies to runtime. IMHO
> > having CONFIG_TIERED_MEMORY is better than using CONFIG_NUMA.
>
> I don't think CONFIG_TIERED_MEMORY bring no much value. It's better
> to use CONFIG_NUMA directly. But this is just my opinion.
I agree. As long as it's always built with CONFIG_NUMA, it's simply
NUMA code. Easy enough to modularize it later if somebody really wants
this to be configurable separately.
On 6/13/22 6:46 PM, Johannes Weiner wrote:
> On Mon, Jun 13, 2022 at 01:30:08PM +0800, Ying Huang wrote:
>> On Mon, 2022-06-13 at 09:01 +0530, Aneesh Kumar K V wrote:
>>> On 6/13/22 8:52 AM, Ying Huang wrote:
>>>> On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
>>>>> +config TIERED_MEMORY
>>>>> + def_bool NUMA
>>>>> +
>>>>
>>>> As Yang pointed out, why not just use CONFIG_NUMA? I suspect the
>>>> added value of CONIFIG_TIRED_MEMORY.
>>>
>>> I decided to use TIERED_MEMORY to bring more clarity. It should be same
>>> now that we have moved CONFIG_MIGRATION dependencies to runtime. IMHO
>>> having CONFIG_TIERED_MEMORY is better than using CONFIG_NUMA.
>>
>> I don't think CONFIG_TIERED_MEMORY bring no much value. It's better
>> to use CONFIG_NUMA directly. But this is just my opinion.
>
> I agree. As long as it's always built with CONFIG_NUMA, it's simply
> NUMA code. Easy enough to modularize it later if somebody really wants
> this to be configurable separately.
I was comparing,
#ifdef CONFIG_TIERED_MEMORY
struct memory_tier {
vs
#ifdef CONFIG_NUMA
struct memory_tier {
I will switch to CONFIG_NUMA in the next update since you are not
finding it beneficial.
-aneesh
Ying Huang <[email protected]> writes:
....
>
>> All those functions are called with memory_tier_lock_held. Infact all
>> list operations requires that lock held. What details do you suggest we
>> document? I can add extra comment to the mutex itself? Adding locking
>> details to all the functions will be duplicating the same details at
>> multiple places?
>
> memory_tier_lock isn't held to call register_memory_tier() in this
> patch. That will cause confusion.
will this help to explain this better
modified mm/memory-tiers.c
@@ -151,6 +151,11 @@ static void insert_memory_tier(struct memory_tier *memtier)
struct list_head *ent;
struct memory_tier *tmp_memtier;
+ if (IS_ENABLED(CONFIG_DEBUG_VM) && !mutex_is_locked(&memory_tier_lock)) {
+ WARN_ON_ONCE(1);
+ return;
+ }
+
list_for_each(ent, &memory_tiers) {
tmp_memtier = list_entry(ent, struct memory_tier, list);
if (tmp_memtier->rank < memtier->rank) {
@@ -811,8 +816,12 @@ static int __init memory_tier_init(void)
/*
* Register only default memory tier to hide all empty
- * memory tier from sysfs.
+ * memory tier from sysfs. Since this is early during
+ * boot, we could avoid holding memtory_tier_lock. But
+ * keep it simple by holding locks. We can add lock
+ * held debug checks in other functions.
*/
+ mutex_lock(&memory_tier_lock);
memtier = register_memory_tier(DEFAULT_MEMORY_TIER,
get_rank_from_tier(DEFAULT_MEMORY_TIER));
@@ -828,6 +837,7 @@ static int __init memory_tier_init(void)
NODE_DATA(node)->memtier = memtier;
node_set(node, memtier->nodelist);
}
+ mutex_unlock(&memory_tier_lock);
migrate_on_reclaim_init();
return 0;
-aneesh
On Mon, 2022-06-13 at 11:18 +0530, Aneesh Kumar K V wrote:
> On 6/13/22 11:03 AM, Ying Huang wrote:
> > On Mon, 2022-06-13 at 09:05 +0530, Aneesh Kumar K V wrote:
> > > On 6/13/22 8:56 AM, Ying Huang wrote:
> > > > On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote:
> > > > > With CONFIG_MIGRATION disabled return EINVAL on write.
> > > > >
> > > > > Signed-off-by: Aneesh Kumar K.V <[email protected]>
> > > > > ---
> > > > > mm/memory-tiers.c | 3 +++
> > > > > 1 file changed, 3 insertions(+)
> > > > >
> > > > > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> > > > > index 9c6b40d7e0bf..c3123a457d90 100644
> > > > > --- a/mm/memory-tiers.c
> > > > > +++ b/mm/memory-tiers.c
> > > > > @@ -105,6 +105,9 @@ static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
> > > > > {
> > > > > ssize_t ret;
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > + if (!IS_ENABLED(CONFIG_MIGRATION))
> > > > > + return -EINVAL;
> > > > > +
> > > >
> > > > How about enclose numa_demotion_enabled_xxx related code with CONFIG_MIGRATION?
> > > >
> > >
> > > IIUC there is a desire to use IS_ENABLED() in the kernel instead of
> > > #ifdef since that helps in more compile time checks. Because there are
> > > no dead codes during compile now with IS_ENABLED().
> >
> > IS_ENABLED() is used to reduce usage of "#ifdef" in ".c" file,
> > especially inside a function. We have good build test coverage with
> > 0Day now.
> >
> > To avoid code size inflate, it's better to use #ifdef CONFIG_MIGRATION.
> >
>
> For a diff like below I am finding IS_ENABLED better.
>
> size memory-tiers.o.isenabled memory-tiers.o
> text data bss dec hex filename
> 4776 989 5 5770 168a memory-tiers.o.isenabled
> 5257 990 5 6252 186c memory-tiers.o
>
>
> modified mm/memory-tiers.c
> @@ -710,12 +710,11 @@ static int __meminit
> migrate_on_reclaim_callback(struct notifier_block *self,
>
> static void __init migrate_on_reclaim_init(void)
> {
> -
> - if (IS_ENABLED(CONFIG_MIGRATION)) {
> +#ifdef CONFIG_MIGRATION
> node_demotion = kcalloc(MAX_NUMNODES, sizeof(struct demotion_nodes),
> GFP_KERNEL);
> WARN_ON(!node_demotion);
> - }
> +#endif
> hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
> }
>
> @@ -844,14 +843,19 @@ static ssize_t numa_demotion_enabled_show(struct
> kobject *kobj,
> numa_demotion_enabled ? "true" : "false");
> }
>
> +#ifdef CONFIG_MIGRATION
> static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
> struct kobj_attribute *attr,
> const char *buf, size_t count)
> {
> - ssize_t ret;
> -
> - if (!IS_ENABLED(CONFIG_MIGRATION))
> - return -EINVAL;
> + return -EINVAL;
> +}
> +#else
> +static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
> + struct kobj_attribute *attr,
> + const char *buf, size_t count)
> +{
> + ssize_t ret;
>
> ret = kstrtobool(buf, &numa_demotion_enabled);
> if (ret)
> @@ -859,6 +863,7 @@ static ssize_t numa_demotion_enabled_store(struct
> kobject *kobj,
>
> return count;
> }
> +#endif
>
> static struct kobj_attribute numa_demotion_enabled_attr =
> __ATTR(demotion_enabled, 0644, numa_demotion_enabled_show,
>
> I also find that #ifdef config not easier to the eyes. If there is a
> large code that we can end up #ifdef out, then it might be worth it.
> IIUC, we might want to keep the establish_migration target to find
> top_tier rank and lower_tier mask. Once we do that only thing that we
> could comment out is the node_demotion sysfs creation and I was
> considering to keep that even if migration is disabled with a write to
> the file returning EINVAL. I could switch that if you strongly feel that
> we should hide node_demotion sysfs file.
Per my understanding, we can enclose most code about
demoting/promoting inside CONFIG_MIGRATION, including
numa/demotion_enabled sysfs interface. In this way, the code size can
be reduced.
Best Regards,
Huang, Ying
>> memory_tier_lock isn't held to call register_memory_tier() in this
>> patch. That will cause confusion.
>
>will this help to explain this better
>modified mm/memory-tiers.c
>@@ -151,6 +151,11 @@ static void insert_memory_tier(struct memory_tier *memtier)
> struct list_head *ent;
> struct memory_tier *tmp_memtier;
>
>+ if (IS_ENABLED(CONFIG_DEBUG_VM) && !mutex_is_locked(&memory_tier_lock)) {
>+ WARN_ON_ONCE(1);
>+ return;
>+ }
Why not just use lockdep here instead?