The current kernel has the basic memory tiering support: Inactive pages on a
higher tier NUMA node can be migrated (demoted) to a lower tier NUMA node to
make room for new allocations on the higher tier NUMA node. Frequently accessed
pages on a lower tier NUMA node can be migrated (promoted) to a higher tier NUMA
node to improve the performance.
In the current kernel, memory tiers are defined implicitly via a demotion path
relationship between NUMA nodes, which is created during the kernel
initialization and updated when a NUMA node is hot-added or hot-removed. The
current implementation puts all nodes with CPU into the highest tier, and builds the
tier hierarchy tier-by-tier by establishing the per-node demotion targets based
on the distances between nodes.
This current memory tier kernel implementation needs to be improved for several
important use cases:
* The current tier initialization code always initializes each memory-only NUMA
node into a lower tier. But a memory-only NUMA node may have a high
performance memory device (e.g. a DRAM-backed memory-only node on a virtual
machine) and that should be put into a higher tier.
* The current tier hierarchy always puts CPU nodes into the top tier. But on a
system with HBM (e.g. GPU memory) devices, these memory-only HBM NUMA nodes
should be in the top tier, and DRAM nodes with CPUs are better to be placed
into the next lower tier.
* Also because the current tier hierarchy always puts CPU nodes into the top
tier, when a CPU is hot-added (or hot-removed) and triggers a memory node from
CPU-less into a CPU node (or vice versa), the memory tier hierarchy gets
changed, even though no memory node is added or removed. This can make the
tier hierarchy unstable and make it difficult to support tier-based memory
accounting.
* A higher tier node can only be demoted to nodes with shortest distance on the
next lower tier as defined by the demotion path, not any other node from any
lower tier. This strict, demotion order does not work in all use
cases (e.g. some use cases may want to allow cross-socket demotion to another
node in the same demotion tier as a fallback when the preferred demotion node
is out of space), and has resulted in the feature request for an interface to
override the system-wide, per-node demotion order from the userspace. This
demotion order is also inconsistent with the page allocation fallback order
when all the nodes in a higher tier are out of space: The page allocation can
fall back to any node from any lower tier, whereas the demotion order doesn't
allow that.
This patch series make the creation of memory tiers explicit under
the control of device driver.
Memory Tier Initialization
==========================
Linux kernel presents memory devices as NUMA nodes and each memory device is of
a specific type. The memory type of a device is represented by its abstract
distance. A memory tier corresponds to a range of abstract distance. This allows
for classifying memory devices with a specific performance range into a memory
tier.
By default, all memory nodes are assigned to the default tier with
abstract distance 512.
A device driver can move its memory nodes from the default tier. For example,
PMEM can move its memory nodes below the default tier, whereas GPU can move its
memory nodes above the default tier.
The kernel initialization code makes the decision on which exact tier a memory
node should be assigned to based on the requests from the device drivers as well
as the memory device hardware information provided by the firmware.
Hot-adding/removing CPUs doesn't affect memory tier hierarchy.
Changes from v12
* Fix kernel crash on module unload
* Address review feedback.
* Add node_random patch to this series based on review feedback
Changes from v11:
* smaller abstract distance imply faster(higher) memory tier.
Changes from v10:
* rename performance level to abstract distance
* Thanks to all the good feedback from Huang, Ying <[email protected]>.
Updated the patchset to cover most of the review feedback.
Changes from v9:
* Use performance level for initializing memory tiers.
Changes from v8:
* Drop the sysfs interface patches and related documentation changes.
Changes from v7:
* Fix kernel crash with demotion.
* Improve documentation.
Changes from v6:
* Drop the usage of rank.
* Address other review feedback.
Changes from v5:
* Remove patch supporting N_MEMORY node removal from memory tiers. memory tiers
are going to be used for features other than demotion. Hence keep all N_MEMORY
nodes in memory tiers irrespective of whether they want to participate in promotion or demotion.
* Add NODE_DATA->memtier
* Rearrage patches to add sysfs files later.
* Add support to create memory tiers from userspace.
* Address other review feedback.
Changes from v4:
* Address review feedback.
* Reverse the meaning of "rank": higher rank value means higher tier.
* Add "/sys/devices/system/memtier/default_tier".
* Add node_is_toptier
v4:
Add support for explicit memory tiers and ranks.
v3:
- Modify patch 1 subject to make it more specific
- Remove /sys/kernel/mm/numa/demotion_targets interface, use
/sys/devices/system/node/demotion_targets instead and make
it writable to override node_states[N_DEMOTION_TARGETS].
- Add support to view per node demotion targets via sysfs
v2:
In v1, only 1st patch of this patch series was sent, which was
implemented to avoid some of the limitations on the demotion
target sharing, however for certain numa topology, the demotion
targets found by that patch was not most optimal, so 1st patch
in this series is modified according to suggestions from Huang
and Baolin. Different examples of demotion list comparasion
between existing implementation and changed implementation can
be found in the commit message of 1st patch.
Aneesh Kumar K.V (8):
mm/demotion: Add support for explicit memory tiers
mm/demotion: Move memory demotion related code
mm/demotion: Add hotplug callbacks to handle new numa node onlined
mm/demotion/dax/kmem: Set node's abstract distance to
MEMTIER_DEFAULT_DAX_ADISTANCE
mm/demotion: Build demotion targets based on explicit memory tiers
mm/demotion: Add pg_data_t member to track node memory tier details
mm/demotion: Update node_is_toptier to work with memory tiers
lib/nodemask: Optimize node_random for nodemask with single NUMA node
Jagdish Gediya (1):
mm/demotion: Demote pages according to allocation fallback order
drivers/dax/kmem.c | 40 ++-
include/linux/memory-tiers.h | 84 +++++
include/linux/migrate.h | 15 -
include/linux/mmzone.h | 3 +
include/linux/node.h | 5 -
lib/nodemask.c | 15 +-
mm/Makefile | 1 +
mm/huge_memory.c | 1 +
mm/memory-tiers.c | 616 +++++++++++++++++++++++++++++++++++
mm/migrate.c | 453 +-------------------------
mm/mprotect.c | 1 +
mm/vmscan.c | 59 +++-
mm/vmstat.c | 4 -
13 files changed, 802 insertions(+), 495 deletions(-)
create mode 100644 include/linux/memory-tiers.h
create mode 100644 mm/memory-tiers.c
--
2.37.1
Also update different helpes to use NODE_DATA()->memtier. Since
node specific memtier can change based on the reassignment of
NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
needs to happen under an rcu read lock or memory_tier_lock.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/mmzone.h | 3 +++
mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------
2 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index aab70355d64f..353812495a70 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -928,6 +928,9 @@ typedef struct pglist_data {
/* Per-node vmstats */
struct per_cpu_nodestat __percpu *per_cpu_nodestats;
atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
+#ifdef CONFIG_NUMA
+ struct memory_tier __rcu *memtier;
+#endif
} pg_data_t;
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 02e514e87d5c..3778ac6a44a1 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -5,6 +5,7 @@
#include <linux/kobject.h>
#include <linux/memory.h>
#include <linux/random.h>
+#include <linux/mmzone.h>
#include <linux/memory-tiers.h>
#include "internal.h"
@@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty
static struct memory_tier *__node_get_memory_tier(int node)
{
- struct memory_dev_type *memtype;
+ pg_data_t *pgdat;
- memtype = node_memory_types[node];
- if (memtype && node_isset(node, memtype->nodes))
- return memtype->memtier;
- return NULL;
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return NULL;
+ /*
+ * Since we hold memory_tier_lock, we can avoid
+ * RCU read locks when accessing the details. No
+ * parallel updates are possible here.
+ */
+ return rcu_dereference_check(pgdat->memtier,
+ lockdep_is_held(&memory_tier_lock));
}
#ifdef CONFIG_MIGRATION
@@ -295,6 +302,8 @@ static struct memory_tier *set_node_memory_tier(int node)
{
struct memory_tier *memtier;
struct memory_dev_type *memtype;
+ pg_data_t *pgdat = NODE_DATA(node);
+
lockdep_assert_held_once(&memory_tier_lock);
@@ -307,6 +316,8 @@ static struct memory_tier *set_node_memory_tier(int node)
memtype = node_memory_types[node];
node_set(node, memtype->nodes);
memtier = find_create_memory_tier(memtype);
+ if (!IS_ERR(memtier))
+ rcu_assign_pointer(pgdat->memtier, memtier);
return memtier;
}
@@ -319,12 +330,25 @@ static void destroy_memory_tier(struct memory_tier *memtier)
static bool clear_node_memory_tier(int node)
{
bool cleared = false;
+ pg_data_t *pgdat;
struct memory_tier *memtier;
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return false;
+
+ /*
+ * Make sure that anybody looking at NODE_DATA who finds
+ * a valid memtier finds memory_dev_types with nodes still
+ * linked to the memtier. We achieve this by waiting for
+ * rcu read section to finish using synchronize_rcu.
+ */
memtier = __node_get_memory_tier(node);
if (memtier) {
struct memory_dev_type *memtype;
+ rcu_assign_pointer(pgdat->memtier, NULL);
+ synchronize_rcu();
memtype = node_memory_types[node];
node_clear(node, memtype->nodes);
if (nodes_empty(memtype->nodes)) {
@@ -422,8 +446,10 @@ static int __init memory_tier_init(void)
panic("%s() failed to register memory tier: %ld\n",
__func__, PTR_ERR(memtier));
- for_each_node_state(node, N_MEMORY)
+ for_each_node_state(node, N_MEMORY) {
__init_node_memory_type(node, default_dram_type);
+ rcu_assign_pointer(NODE_DATA(node)->memtier, memtier);
+ }
mutex_unlock(&memory_tier_lock);
#ifdef CONFIG_MIGRATION
--
2.37.1
The most common case for certain node_random usage (demotion nodemask) is with
nodemask weight 1. We can avoid calling get_random_init() in that case and
always return the only node set in the nodemask.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
lib/nodemask.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/lib/nodemask.c b/lib/nodemask.c
index e22647f5181b..c91a6b0404a5 100644
--- a/lib/nodemask.c
+++ b/lib/nodemask.c
@@ -20,12 +20,21 @@ EXPORT_SYMBOL(__next_node_in);
*/
int node_random(const nodemask_t *maskp)
{
- int w, bit = NUMA_NO_NODE;
+ int w, bit;
w = nodes_weight(*maskp);
- if (w)
+ switch (w) {
+ case 0:
+ bit = NUMA_NO_NODE;
+ break;
+ case 1:
+ bit = __first_node(maskp);
+ break;
+ default:
bit = bitmap_ord_to_pos(maskp->bits,
- get_random_int() % w, MAX_NUMNODES);
+ get_random_int() % w, MAX_NUMNODES);
+ break;
+ }
return bit;
}
#endif
--
2.37.1
With memory tiers support we can have memory only NUMA nodes
in the top tier from which we want to avoid promotion tracking NUMA
faults. Update node_is_toptier to work with memory tiers.
All NUMA nodes are by default top tier nodes. With lower memory
tiers added we consider all memory tiers above a memory tier having
CPU NUMA nodes as a top memory tier
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/memory-tiers.h | 11 +++++++++
include/linux/node.h | 5 ----
mm/huge_memory.c | 1 +
mm/memory-tiers.c | 46 ++++++++++++++++++++++++++++++++++++
mm/migrate.c | 1 +
mm/mprotect.c | 1 +
6 files changed, 60 insertions(+), 5 deletions(-)
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 341ba8082e05..0bdd5955a5e2 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -35,6 +35,7 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype);
#ifdef CONFIG_MIGRATION
int next_demotion_node(int node);
void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
+bool node_is_toptier(int node);
#else
static inline int next_demotion_node(int node)
{
@@ -45,6 +46,11 @@ static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *target
{
*targets = NODE_MASK_NONE;
}
+
+static inline bool node_is_toptier(int node)
+{
+ return true;
+}
#endif
#else
@@ -69,5 +75,10 @@ static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *target
{
*targets = NODE_MASK_NONE;
}
+
+static inline bool node_is_toptier(int node)
+{
+ return true;
+}
#endif /* CONFIG_NUMA */
#endif /* _LINUX_MEMORY_TIERS_H */
diff --git a/include/linux/node.h b/include/linux/node.h
index 40d641a8bfb0..9ec680dd607f 100644
--- a/include/linux/node.h
+++ b/include/linux/node.h
@@ -185,9 +185,4 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,
#define to_node(device) container_of(device, struct node, dev)
-static inline bool node_is_toptier(int node)
-{
- return node_state(node, N_CPU);
-}
-
#endif /* _LINUX_NODE_H_ */
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 15965084816d..524498061e7c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -36,6 +36,7 @@
#include <linux/numa.h>
#include <linux/page_owner.h>
#include <linux/sched/sysctl.h>
+#include <linux/memory-tiers.h>
#include <asm/tlb.h>
#include <asm/pgalloc.h>
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 925d7168e825..ea5c04f62170 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -33,6 +33,7 @@ static LIST_HEAD(memory_tiers);
static struct memory_dev_type *node_memory_types[MAX_NUMNODES];
static struct memory_dev_type *default_dram_type;
#ifdef CONFIG_MIGRATION
+static int top_tier_adistance;
/*
* node_demotion[] examples:
*
@@ -154,6 +155,31 @@ static struct memory_tier *__node_get_memory_tier(int node)
}
#ifdef CONFIG_MIGRATION
+bool node_is_toptier(int node)
+{
+ bool toptier;
+ pg_data_t *pgdat;
+ struct memory_tier *memtier;
+
+ pgdat = NODE_DATA(node);
+ if (!pgdat)
+ return false;
+
+ rcu_read_lock();
+ memtier = rcu_dereference(pgdat->memtier);
+ if (!memtier) {
+ toptier = true;
+ goto out;
+ }
+ if (memtier->adistance_start < top_tier_adistance)
+ toptier = true;
+ else
+ toptier = false;
+out:
+ rcu_read_unlock();
+ return toptier;
+}
+
void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
{
struct memory_tier *memtier;
@@ -311,6 +337,26 @@ static void establish_demotion_targets(void)
}
} while (1);
}
+ /*
+ * Promotion is allowed from a memory tier to higher
+ * memory tier only if the memory tier doesn't include
+ * compute. We want to skip promotion from a memory tier,
+ * if any node that is part of the memory tier have CPUs.
+ * Once we detect such a memory tier, we consider that tier
+ * as top tiper from which promotion is not allowed.
+ */
+ list_for_each_entry_reverse(memtier, &memory_tiers, list) {
+ tier_nodes = get_memtier_nodemask(memtier);
+ nodes_and(tier_nodes, node_states[N_CPU], tier_nodes);
+ if (!nodes_empty(tier_nodes)) {
+ /*
+ * abstract distance below the max value of this memtier
+ * is considered toptier.
+ */
+ top_tier_adistance = memtier->adistance_start + MEMTIER_CHUNK_SIZE;
+ break;
+ }
+ }
/*
* Now build the lower_tier mask for each node collecting node mask from
* all memory tier below it. This allows us to fallback demotion page
diff --git a/mm/migrate.c b/mm/migrate.c
index 45290ddd3806..e7f3f52596c1 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -50,6 +50,7 @@
#include <linux/memory.h>
#include <linux/random.h>
#include <linux/sched/sysctl.h>
+#include <linux/memory-tiers.h>
#include <asm/tlbflush.h>
diff --git a/mm/mprotect.c b/mm/mprotect.c
index ba5592655ee3..92a2fc0fa88b 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -31,6 +31,7 @@
#include <linux/pgtable.h>
#include <linux/sched/sysctl.h>
#include <linux/userfaultfd_k.h>
+#include <linux/memory-tiers.h>
#include <asm/cacheflush.h>
#include <asm/mmu_context.h>
#include <asm/tlbflush.h>
--
2.37.1
In the current kernel, memory tiers are defined implicitly via a demotion path
relationship between NUMA nodes, which is created during the kernel
initialization and updated when a NUMA node is hot-added or hot-removed. The
current implementation puts all nodes with CPU into the highest tier, and builds
the tier hierarchy by establishing the per-node demotion targets based on the
distances between nodes.
This current memory tier kernel implementation needs to be improved for several
important use cases,
The current tier initialization code always initializes each memory-only NUMA
node into a lower tier. But a memory-only NUMA node may have a high performance
memory device (e.g. a DRAM-backed memory-only node on a virtual machine) that
should be put into a higher tier.
The current tier hierarchy always puts CPU nodes into the top tier. But on a
system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices
should be in the top tier, and DRAM nodes with CPUs are better to be placed into
the next lower tier.
With current kernel higher tier node can only be demoted to nodes with shortest
distance on the next lower tier as defined by the demotion path, not any other
node from any lower tier. This strict, demotion order does not work in all use
cases (e.g. some use cases may want to allow cross-socket demotion to another
node in the same demotion tier as a fallback when the preferred demotion node is
out of space), This demotion order is also inconsistent with the page allocation
fallback order when all the nodes in a higher tier are out of space: The page
allocation can fall back to any node from any lower tier, whereas the demotion
order doesn't allow that.
This patch series address the above by defining memory tiers explicitly.
Linux kernel presents memory devices as NUMA nodes and each memory device is of
a specific type. The memory type of a device is represented by its abstract
distance. A memory tier corresponds to a range of abstract distance. This allows
for classifying memory devices with a specific performance range into a memory
tier.
This patch configures the range/chunk size to be 128. The default DRAM abstract
distance is 512. We can have 4 memory tiers below the default DRAM with abstract
distance range 0 - 127, 127 - 255, 256- 383, 384 - 511. Faster memory devices
can be placed in these faster(higher) memory tiers. Slower memory devices like
persistent memory will have abstract distance higher than the default DRAM
level.
Signed-off-by: Aneesh Kumar K.V <[email protected]>
---
include/linux/memory-tiers.h | 15 +++++
mm/Makefile | 1 +
mm/memory-tiers.c | 107 +++++++++++++++++++++++++++++++++++
3 files changed, 123 insertions(+)
create mode 100644 include/linux/memory-tiers.h
create mode 100644 mm/memory-tiers.c
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
new file mode 100644
index 000000000000..bc7c1b799bef
--- /dev/null
+++ b/include/linux/memory-tiers.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_TIERS_H
+#define _LINUX_MEMORY_TIERS_H
+
+/*
+ * Each tier cover a abstrace distance chunk size of 128
+ */
+#define MEMTIER_CHUNK_BITS 7
+#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS)
+/*
+ * Smaller abstract distance value imply faster(higher) memory tiers.
+ */
+#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE)
+
+#endif /* _LINUX_MEMORY_TIERS_H */
diff --git a/mm/Makefile b/mm/Makefile
index 6f9ffa968a1a..d30acebc2164 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_NUMA) += memory-tiers.o
obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
new file mode 100644
index 000000000000..78b311d9bde9
--- /dev/null
+++ b/mm/memory-tiers.c
@@ -0,0 +1,107 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+#include <linux/nodemask.h>
+#include <linux/slab.h>
+#include <linux/lockdep.h>
+#include <linux/memory-tiers.h>
+
+struct memory_tier {
+ /* hierarchy of memory tiers */
+ struct list_head list;
+ /* list of all memory types part of this tier */
+ struct list_head memory_types;
+ /*
+ * start value of abstract distance. memory tier maps
+ * an abstract distance range,
+ * adistance_start .. adistance_start + MEMTIER_CHUNK_SIZE
+ */
+ int adistance_start;
+};
+
+struct memory_dev_type {
+ /* list of memory types that are part of same tier as this type */
+ struct list_head tier_sibiling;
+ /* abstract distance for this specific memory type */
+ int adistance;
+ /* Nodes of same abstract distance */
+ nodemask_t nodes;
+ struct memory_tier *memtier;
+};
+
+static DEFINE_MUTEX(memory_tier_lock);
+static LIST_HEAD(memory_tiers);
+static struct memory_dev_type *node_memory_types[MAX_NUMNODES];
+/*
+ * For now let's have 4 memory tier below default DRAM tier.
+ */
+static struct memory_dev_type default_dram_type = {
+ .adistance = MEMTIER_ADISTANCE_DRAM,
+ .tier_sibiling = LIST_HEAD_INIT(default_dram_type.tier_sibiling),
+};
+
+static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtype)
+{
+ bool found_slot = false;
+ struct memory_tier *memtier, *new_memtier;
+ int adistance = memtype->adistance;
+ unsigned int memtier_adistance_chunk_size = MEMTIER_CHUNK_SIZE;
+
+ lockdep_assert_held_once(&memory_tier_lock);
+
+ /*
+ * If the memtype is already part of a memory tier,
+ * just return that.
+ */
+ if (memtype->memtier)
+ return memtype->memtier;
+
+ adistance = round_down(adistance, memtier_adistance_chunk_size);
+ list_for_each_entry(memtier, &memory_tiers, list) {
+ if (adistance == memtier->adistance_start) {
+ memtype->memtier = memtier;
+ list_add(&memtype->tier_sibiling, &memtier->memory_types);
+ return memtier;
+ } else if (adistance < memtier->adistance_start) {
+ found_slot = true;
+ break;
+ }
+ }
+
+ new_memtier = kmalloc(sizeof(struct memory_tier), GFP_KERNEL);
+ if (!new_memtier)
+ return ERR_PTR(-ENOMEM);
+
+ new_memtier->adistance_start = adistance;
+ INIT_LIST_HEAD(&new_memtier->list);
+ INIT_LIST_HEAD(&new_memtier->memory_types);
+ if (found_slot)
+ list_add_tail(&new_memtier->list, &memtier->list);
+ else
+ list_add_tail(&new_memtier->list, &memory_tiers);
+ memtype->memtier = new_memtier;
+ list_add(&memtype->tier_sibiling, &new_memtier->memory_types);
+ return new_memtier;
+}
+
+static int __init memory_tier_init(void)
+{
+ int node;
+ struct memory_tier *memtier;
+
+ mutex_lock(&memory_tier_lock);
+ /* CPU only nodes are not part of memory tiers. */
+ default_dram_type.nodes = node_states[N_MEMORY];
+
+ memtier = find_create_memory_tier(&default_dram_type);
+ if (IS_ERR(memtier))
+ panic("%s() failed to register memory tier: %ld\n",
+ __func__, PTR_ERR(memtier));
+
+ for_each_node_state(node, N_MEMORY)
+ node_memory_types[node] = &default_dram_type;
+
+ mutex_unlock(&memory_tier_lock);
+
+ return 0;
+}
+subsys_initcall(memory_tier_init);
--
2.37.1
"Aneesh Kumar K.V" <[email protected]> writes:
> In the current kernel, memory tiers are defined implicitly via a demotion path
> relationship between NUMA nodes, which is created during the kernel
> initialization and updated when a NUMA node is hot-added or hot-removed. The
> current implementation puts all nodes with CPU into the highest tier, and builds
> the tier hierarchy by establishing the per-node demotion targets based on the
> distances between nodes.
>
> This current memory tier kernel implementation needs to be improved for several
> important use cases,
>
> The current tier initialization code always initializes each memory-only NUMA
> node into a lower tier. But a memory-only NUMA node may have a high performance
> memory device (e.g. a DRAM-backed memory-only node on a virtual machine) that
> should be put into a higher tier.
>
> The current tier hierarchy always puts CPU nodes into the top tier. But on a
> system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices
> should be in the top tier, and DRAM nodes with CPUs are better to be placed into
> the next lower tier.
>
> With current kernel higher tier node can only be demoted to nodes with shortest
> distance on the next lower tier as defined by the demotion path, not any other
> node from any lower tier. This strict, demotion order does not work in all use
> cases (e.g. some use cases may want to allow cross-socket demotion to another
> node in the same demotion tier as a fallback when the preferred demotion node is
> out of space), This demotion order is also inconsistent with the page allocation
> fallback order when all the nodes in a higher tier are out of space: The page
> allocation can fall back to any node from any lower tier, whereas the demotion
> order doesn't allow that.
>
> This patch series address the above by defining memory tiers explicitly.
>
> Linux kernel presents memory devices as NUMA nodes and each memory device is of
> a specific type. The memory type of a device is represented by its abstract
> distance. A memory tier corresponds to a range of abstract distance. This allows
> for classifying memory devices with a specific performance range into a memory
> tier.
>
> This patch configures the range/chunk size to be 128. The default DRAM abstract
> distance is 512. We can have 4 memory tiers below the default DRAM with abstract
> distance range 0 - 127, 127 - 255, 256- 383, 384 - 511. Faster memory devices
> can be placed in these faster(higher) memory tiers. Slower memory devices like
> persistent memory will have abstract distance higher than the default DRAM
> level.
>
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> include/linux/memory-tiers.h | 15 +++++
> mm/Makefile | 1 +
> mm/memory-tiers.c | 107 +++++++++++++++++++++++++++++++++++
> 3 files changed, 123 insertions(+)
> create mode 100644 include/linux/memory-tiers.h
> create mode 100644 mm/memory-tiers.c
>
> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
> new file mode 100644
> index 000000000000..bc7c1b799bef
> --- /dev/null
> +++ b/include/linux/memory-tiers.h
> @@ -0,0 +1,15 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _LINUX_MEMORY_TIERS_H
> +#define _LINUX_MEMORY_TIERS_H
> +
> +/*
> + * Each tier cover a abstrace distance chunk size of 128
> + */
> +#define MEMTIER_CHUNK_BITS 7
> +#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS)
> +/*
> + * Smaller abstract distance value imply faster(higher) memory tiers.
> + */
> +#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE)
> +
> +#endif /* _LINUX_MEMORY_TIERS_H */
> diff --git a/mm/Makefile b/mm/Makefile
> index 6f9ffa968a1a..d30acebc2164 100644
> --- a/mm/Makefile
> +++ b/mm/Makefile
> @@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
> obj-$(CONFIG_FAILSLAB) += failslab.o
> obj-$(CONFIG_MEMTEST) += memtest.o
> obj-$(CONFIG_MIGRATION) += migrate.o
> +obj-$(CONFIG_NUMA) += memory-tiers.o
> obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
> obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
> obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> new file mode 100644
> index 000000000000..78b311d9bde9
> --- /dev/null
> +++ b/mm/memory-tiers.c
> @@ -0,0 +1,107 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <linux/types.h>
> +#include <linux/nodemask.h>
> +#include <linux/slab.h>
> +#include <linux/lockdep.h>
> +#include <linux/memory-tiers.h>
> +
> +struct memory_tier {
> + /* hierarchy of memory tiers */
> + struct list_head list;
> + /* list of all memory types part of this tier */
> + struct list_head memory_types;
> + /*
> + * start value of abstract distance. memory tier maps
> + * an abstract distance range,
> + * adistance_start .. adistance_start + MEMTIER_CHUNK_SIZE
> + */
> + int adistance_start;
> +};
> +
> +struct memory_dev_type {
> + /* list of memory types that are part of same tier as this type */
> + struct list_head tier_sibiling;
> + /* abstract distance for this specific memory type */
> + int adistance;
> + /* Nodes of same abstract distance */
> + nodemask_t nodes;
> + struct memory_tier *memtier;
> +};
> +
> +static DEFINE_MUTEX(memory_tier_lock);
> +static LIST_HEAD(memory_tiers);
> +static struct memory_dev_type *node_memory_types[MAX_NUMNODES];
> +/*
> + * For now let's have 4 memory tier below default DRAM tier.
> + */
> +static struct memory_dev_type default_dram_type = {
> + .adistance = MEMTIER_ADISTANCE_DRAM,
> + .tier_sibiling = LIST_HEAD_INIT(default_dram_type.tier_sibiling),
> +};
> +
> +static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtype)
> +{
> + bool found_slot = false;
> + struct memory_tier *memtier, *new_memtier;
> + int adistance = memtype->adistance;
> + unsigned int memtier_adistance_chunk_size = MEMTIER_CHUNK_SIZE;
> +
> + lockdep_assert_held_once(&memory_tier_lock);
> +
> + /*
> + * If the memtype is already part of a memory tier,
> + * just return that.
> + */
> + if (memtype->memtier)
> + return memtype->memtier;
> +
> + adistance = round_down(adistance, memtier_adistance_chunk_size);
> + list_for_each_entry(memtier, &memory_tiers, list) {
> + if (adistance == memtier->adistance_start) {
> + memtype->memtier = memtier;
> + list_add(&memtype->tier_sibiling, &memtier->memory_types);
> + return memtier;
> + } else if (adistance < memtier->adistance_start) {
> + found_slot = true;
> + break;
> + }
> + }
> +
> + new_memtier = kmalloc(sizeof(struct memory_tier), GFP_KERNEL);
> + if (!new_memtier)
> + return ERR_PTR(-ENOMEM);
> +
> + new_memtier->adistance_start = adistance;
> + INIT_LIST_HEAD(&new_memtier->list);
> + INIT_LIST_HEAD(&new_memtier->memory_types);
> + if (found_slot)
> + list_add_tail(&new_memtier->list, &memtier->list);
> + else
> + list_add_tail(&new_memtier->list, &memory_tiers);
> + memtype->memtier = new_memtier;
> + list_add(&memtype->tier_sibiling, &new_memtier->memory_types);
> + return new_memtier;
> +}
> +
> +static int __init memory_tier_init(void)
> +{
> + int node;
> + struct memory_tier *memtier;
> +
> + mutex_lock(&memory_tier_lock);
> + /* CPU only nodes are not part of memory tiers. */
> + default_dram_type.nodes = node_states[N_MEMORY];
> +
> + memtier = find_create_memory_tier(&default_dram_type);
> + if (IS_ERR(memtier))
> + panic("%s() failed to register memory tier: %ld\n",
> + __func__, PTR_ERR(memtier));
> +
> + for_each_node_state(node, N_MEMORY)
> + node_memory_types[node] = &default_dram_type;
Although not absolutely necessary, it seems better to set
node_memory_types[] before adding nodes to the memory type and adding
the memory type to the memory tier.
Best Regards,
Huang, Ying
> +
> + mutex_unlock(&memory_tier_lock);
> +
> + return 0;
> +}
> +subsys_initcall(memory_tier_init);
"Aneesh Kumar K.V" <[email protected]> writes:
> Also update different helpes to use NODE_DATA()->memtier. Since
> node specific memtier can change based on the reassignment of
> NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
> needs to happen under an rcu read lock or memory_tier_lock.
>
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> include/linux/mmzone.h | 3 +++
> mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------
> 2 files changed, 35 insertions(+), 6 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index aab70355d64f..353812495a70 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -928,6 +928,9 @@ typedef struct pglist_data {
> /* Per-node vmstats */
> struct per_cpu_nodestat __percpu *per_cpu_nodestats;
> atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
> +#ifdef CONFIG_NUMA
> + struct memory_tier __rcu *memtier;
> +#endif
> } pg_data_t;
>
> #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 02e514e87d5c..3778ac6a44a1 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -5,6 +5,7 @@
> #include <linux/kobject.h>
> #include <linux/memory.h>
> #include <linux/random.h>
> +#include <linux/mmzone.h>
> #include <linux/memory-tiers.h>
>
> #include "internal.h"
> @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty
>
> static struct memory_tier *__node_get_memory_tier(int node)
> {
> - struct memory_dev_type *memtype;
> + pg_data_t *pgdat;
>
> - memtype = node_memory_types[node];
> - if (memtype && node_isset(node, memtype->nodes))
> - return memtype->memtier;
> - return NULL;
> + pgdat = NODE_DATA(node);
> + if (!pgdat)
> + return NULL;
> + /*
> + * Since we hold memory_tier_lock, we can avoid
> + * RCU read locks when accessing the details. No
> + * parallel updates are possible here.
> + */
> + return rcu_dereference_check(pgdat->memtier,
> + lockdep_is_held(&memory_tier_lock));
> }
>
> #ifdef CONFIG_MIGRATION
> @@ -295,6 +302,8 @@ static struct memory_tier *set_node_memory_tier(int node)
> {
> struct memory_tier *memtier;
> struct memory_dev_type *memtype;
> + pg_data_t *pgdat = NODE_DATA(node);
> +
>
> lockdep_assert_held_once(&memory_tier_lock);
>
> @@ -307,6 +316,8 @@ static struct memory_tier *set_node_memory_tier(int node)
> memtype = node_memory_types[node];
> node_set(node, memtype->nodes);
> memtier = find_create_memory_tier(memtype);
> + if (!IS_ERR(memtier))
> + rcu_assign_pointer(pgdat->memtier, memtier);
> return memtier;
> }
>
> @@ -319,12 +330,25 @@ static void destroy_memory_tier(struct memory_tier *memtier)
> static bool clear_node_memory_tier(int node)
> {
> bool cleared = false;
> + pg_data_t *pgdat;
> struct memory_tier *memtier;
>
> + pgdat = NODE_DATA(node);
> + if (!pgdat)
> + return false;
> +
> + /*
> + * Make sure that anybody looking at NODE_DATA who finds
> + * a valid memtier finds memory_dev_types with nodes still
> + * linked to the memtier. We achieve this by waiting for
> + * rcu read section to finish using synchronize_rcu.
> + */
The synchronize_rcu() is also needed because we may free the memory
tier.
Best Regards,
Huang, Ying
> memtier = __node_get_memory_tier(node);
> if (memtier) {
> struct memory_dev_type *memtype;
>
> + rcu_assign_pointer(pgdat->memtier, NULL);
> + synchronize_rcu();
> memtype = node_memory_types[node];
> node_clear(node, memtype->nodes);
> if (nodes_empty(memtype->nodes)) {
> @@ -422,8 +446,10 @@ static int __init memory_tier_init(void)
> panic("%s() failed to register memory tier: %ld\n",
> __func__, PTR_ERR(memtier));
>
> - for_each_node_state(node, N_MEMORY)
> + for_each_node_state(node, N_MEMORY) {
> __init_node_memory_type(node, default_dram_type);
> + rcu_assign_pointer(NODE_DATA(node)->memtier, memtier);
> + }
>
> mutex_unlock(&memory_tier_lock);
> #ifdef CONFIG_MIGRATION
"Aneesh Kumar K.V" <[email protected]> writes:
> The most common case for certain node_random usage (demotion nodemask) is with
> nodemask weight 1. We can avoid calling get_random_init() in that case and
> always return the only node set in the nodemask.
I think that this patch can sit between [5/9] and [6/9], just after it
is used.
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> lib/nodemask.c | 15 ++++++++++++---
> 1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/lib/nodemask.c b/lib/nodemask.c
> index e22647f5181b..c91a6b0404a5 100644
> --- a/lib/nodemask.c
> +++ b/lib/nodemask.c
> @@ -20,12 +20,21 @@ EXPORT_SYMBOL(__next_node_in);
> */
> int node_random(const nodemask_t *maskp)
> {
> - int w, bit = NUMA_NO_NODE;
> + int w, bit;
>
> w = nodes_weight(*maskp);
> - if (w)
> + switch (w) {
> + case 0:
> + bit = NUMA_NO_NODE;
> + break;
> + case 1:
> + bit = __first_node(maskp);
Per my understanding, first_node() is the formal API and we should use
that? Just like we use nodes_weight() instead of __nodes_weight().
Best Regards,
Huang, Ying
> + break;
> + default:
> bit = bitmap_ord_to_pos(maskp->bits,
> - get_random_int() % w, MAX_NUMNODES);
> + get_random_int() % w, MAX_NUMNODES);
> + break;
> + }
> return bit;
> }
> #endif
"Aneesh Kumar K.V" <[email protected]> writes:
> Also update different helpes to use NODE_DATA()->memtier. Since
> node specific memtier can change based on the reassignment of
> NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
> needs to happen under an rcu read lock or memory_tier_lock.
>
> Signed-off-by: Aneesh Kumar K.V <[email protected]>
> ---
> include/linux/mmzone.h | 3 +++
> mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------
> 2 files changed, 35 insertions(+), 6 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index aab70355d64f..353812495a70 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -928,6 +928,9 @@ typedef struct pglist_data {
> /* Per-node vmstats */
> struct per_cpu_nodestat __percpu *per_cpu_nodestats;
> atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
> +#ifdef CONFIG_NUMA
> + struct memory_tier __rcu *memtier;
> +#endif
> } pg_data_t;
>
> #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 02e514e87d5c..3778ac6a44a1 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -5,6 +5,7 @@
> #include <linux/kobject.h>
> #include <linux/memory.h>
> #include <linux/random.h>
> +#include <linux/mmzone.h>
> #include <linux/memory-tiers.h>
>
> #include "internal.h"
> @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty
>
> static struct memory_tier *__node_get_memory_tier(int node)
> {
> - struct memory_dev_type *memtype;
> + pg_data_t *pgdat;
>
> - memtype = node_memory_types[node];
> - if (memtype && node_isset(node, memtype->nodes))
> - return memtype->memtier;
> - return NULL;
After adding pgdat->memtier, it appears there's unnecessary to keep
memtype->memtier?
Best Regards,
Huang, Ying
> + pgdat = NODE_DATA(node);
> + if (!pgdat)
> + return NULL;
> + /*
> + * Since we hold memory_tier_lock, we can avoid
> + * RCU read locks when accessing the details. No
> + * parallel updates are possible here.
> + */
> + return rcu_dereference_check(pgdat->memtier,
> + lockdep_is_held(&memory_tier_lock));
> }
>
> #ifdef CONFIG_MIGRATION
> @@ -295,6 +302,8 @@ static struct memory_tier *set_node_memory_tier(int node)
> {
> struct memory_tier *memtier;
> struct memory_dev_type *memtype;
> + pg_data_t *pgdat = NODE_DATA(node);
> +
>
> lockdep_assert_held_once(&memory_tier_lock);
>
> @@ -307,6 +316,8 @@ static struct memory_tier *set_node_memory_tier(int node)
> memtype = node_memory_types[node];
> node_set(node, memtype->nodes);
> memtier = find_create_memory_tier(memtype);
> + if (!IS_ERR(memtier))
> + rcu_assign_pointer(pgdat->memtier, memtier);
> return memtier;
> }
>
> @@ -319,12 +330,25 @@ static void destroy_memory_tier(struct memory_tier *memtier)
> static bool clear_node_memory_tier(int node)
> {
> bool cleared = false;
> + pg_data_t *pgdat;
> struct memory_tier *memtier;
>
> + pgdat = NODE_DATA(node);
> + if (!pgdat)
> + return false;
> +
> + /*
> + * Make sure that anybody looking at NODE_DATA who finds
> + * a valid memtier finds memory_dev_types with nodes still
> + * linked to the memtier. We achieve this by waiting for
> + * rcu read section to finish using synchronize_rcu.
> + */
> memtier = __node_get_memory_tier(node);
> if (memtier) {
> struct memory_dev_type *memtype;
>
> + rcu_assign_pointer(pgdat->memtier, NULL);
> + synchronize_rcu();
> memtype = node_memory_types[node];
> node_clear(node, memtype->nodes);
> if (nodes_empty(memtype->nodes)) {
> @@ -422,8 +446,10 @@ static int __init memory_tier_init(void)
> panic("%s() failed to register memory tier: %ld\n",
> __func__, PTR_ERR(memtier));
>
> - for_each_node_state(node, N_MEMORY)
> + for_each_node_state(node, N_MEMORY) {
> __init_node_memory_type(node, default_dram_type);
> + rcu_assign_pointer(NODE_DATA(node)->memtier, memtier);
> + }
>
> mutex_unlock(&memory_tier_lock);
> #ifdef CONFIG_MIGRATION
On 8/9/22 10:51 AM, Huang, Ying wrote:
> "Aneesh Kumar K.V" <[email protected]> writes:
>
>> Also update different helpes to use NODE_DATA()->memtier. Since
>> node specific memtier can change based on the reassignment of
>> NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
>> needs to happen under an rcu read lock or memory_tier_lock.
>>
>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>> ---
>> include/linux/mmzone.h | 3 +++
>> mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------
>> 2 files changed, 35 insertions(+), 6 deletions(-)
>>
>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>> index aab70355d64f..353812495a70 100644
>> --- a/include/linux/mmzone.h
>> +++ b/include/linux/mmzone.h
>> @@ -928,6 +928,9 @@ typedef struct pglist_data {
>> /* Per-node vmstats */
>> struct per_cpu_nodestat __percpu *per_cpu_nodestats;
>> atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
>> +#ifdef CONFIG_NUMA
>> + struct memory_tier __rcu *memtier;
>> +#endif
>> } pg_data_t;
>>
>> #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>> index 02e514e87d5c..3778ac6a44a1 100644
>> --- a/mm/memory-tiers.c
>> +++ b/mm/memory-tiers.c
>> @@ -5,6 +5,7 @@
>> #include <linux/kobject.h>
>> #include <linux/memory.h>
>> #include <linux/random.h>
>> +#include <linux/mmzone.h>
>> #include <linux/memory-tiers.h>
>>
>> #include "internal.h"
>> @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty
>>
>> static struct memory_tier *__node_get_memory_tier(int node)
>> {
>> - struct memory_dev_type *memtype;
>> + pg_data_t *pgdat;
>>
>> - memtype = node_memory_types[node];
>> - if (memtype && node_isset(node, memtype->nodes))
>> - return memtype->memtier;
>> - return NULL;
>
> After adding pgdat->memtier, it appears there's unnecessary to keep
> memtype->memtier?
>
It do simplify find_create_memory_tier() where I use if (memtype->memtier)
to check whether the memtype is already added to a memory tier. I could switch
that to list_empty(memtype->tier_sibiling). But I felt the current one is much
cleaner
.
-aneesh
On 8/9/22 8:43 AM, Huang, Ying wrote:
> "Aneesh Kumar K.V" <[email protected]> writes:
>
>> The most common case for certain node_random usage (demotion nodemask) is with
>> nodemask weight 1. We can avoid calling get_random_init() in that case and
>> always return the only node set in the nodemask.
>
> I think that this patch can sit between [5/9] and [6/9], just after it
> is used.
>
>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>> ---
>> lib/nodemask.c | 15 ++++++++++++---
>> 1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/lib/nodemask.c b/lib/nodemask.c
>> index e22647f5181b..c91a6b0404a5 100644
>> --- a/lib/nodemask.c
>> +++ b/lib/nodemask.c
>> @@ -20,12 +20,21 @@ EXPORT_SYMBOL(__next_node_in);
>> */
>> int node_random(const nodemask_t *maskp)
>> {
>> - int w, bit = NUMA_NO_NODE;
>> + int w, bit;
>>
>> w = nodes_weight(*maskp);
>> - if (w)
>> + switch (w) {
>> + case 0:
>> + bit = NUMA_NO_NODE;
>> + break;
>> + case 1:
>> + bit = __first_node(maskp);
>
> Per my understanding, first_node() is the formal API and we should use
> that? Just like we use nodes_weight() instead of __nodes_weight().
>
updated.
-aneesh
On 8/9/22 7:28 AM, Huang, Ying wrote:
> "Aneesh Kumar K.V" <[email protected]> writes:
>
>> In the current kernel, memory tiers are defined implicitly via a demotion path
>> relationship between NUMA nodes, which is created during the kernel
>> initialization and updated when a NUMA node is hot-added or hot-removed. The
>> current implementation puts all nodes with CPU into the highest tier, and builds
>> the tier hierarchy by establishing the per-node demotion targets based on the
>> distances between nodes.
>>
>> This current memory tier kernel implementation needs to be improved for several
>> important use cases,
>>
>> The current tier initialization code always initializes each memory-only NUMA
>> node into a lower tier. But a memory-only NUMA node may have a high performance
>> memory device (e.g. a DRAM-backed memory-only node on a virtual machine) that
>> should be put into a higher tier.
>>
>> The current tier hierarchy always puts CPU nodes into the top tier. But on a
>> system with HBM or GPU devices, the memory-only NUMA nodes mapping these devices
>> should be in the top tier, and DRAM nodes with CPUs are better to be placed into
>> the next lower tier.
>>
>> With current kernel higher tier node can only be demoted to nodes with shortest
>> distance on the next lower tier as defined by the demotion path, not any other
>> node from any lower tier. This strict, demotion order does not work in all use
>> cases (e.g. some use cases may want to allow cross-socket demotion to another
>> node in the same demotion tier as a fallback when the preferred demotion node is
>> out of space), This demotion order is also inconsistent with the page allocation
>> fallback order when all the nodes in a higher tier are out of space: The page
>> allocation can fall back to any node from any lower tier, whereas the demotion
>> order doesn't allow that.
>>
>> This patch series address the above by defining memory tiers explicitly.
>>
>> Linux kernel presents memory devices as NUMA nodes and each memory device is of
>> a specific type. The memory type of a device is represented by its abstract
>> distance. A memory tier corresponds to a range of abstract distance. This allows
>> for classifying memory devices with a specific performance range into a memory
>> tier.
>>
>> This patch configures the range/chunk size to be 128. The default DRAM abstract
>> distance is 512. We can have 4 memory tiers below the default DRAM with abstract
>> distance range 0 - 127, 127 - 255, 256- 383, 384 - 511. Faster memory devices
>> can be placed in these faster(higher) memory tiers. Slower memory devices like
>> persistent memory will have abstract distance higher than the default DRAM
>> level.
>>
>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>> ---
>> include/linux/memory-tiers.h | 15 +++++
>> mm/Makefile | 1 +
>> mm/memory-tiers.c | 107 +++++++++++++++++++++++++++++++++++
>> 3 files changed, 123 insertions(+)
>> create mode 100644 include/linux/memory-tiers.h
>> create mode 100644 mm/memory-tiers.c
>>
>> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
>> new file mode 100644
>> index 000000000000..bc7c1b799bef
>> --- /dev/null
>> +++ b/include/linux/memory-tiers.h
>> @@ -0,0 +1,15 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _LINUX_MEMORY_TIERS_H
>> +#define _LINUX_MEMORY_TIERS_H
>> +
>> +/*
>> + * Each tier cover a abstrace distance chunk size of 128
>> + */
>> +#define MEMTIER_CHUNK_BITS 7
>> +#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS)
>> +/*
>> + * Smaller abstract distance value imply faster(higher) memory tiers.
>> + */
>> +#define MEMTIER_ADISTANCE_DRAM (4 * MEMTIER_CHUNK_SIZE)
>> +
>> +#endif /* _LINUX_MEMORY_TIERS_H */
>> diff --git a/mm/Makefile b/mm/Makefile
>> index 6f9ffa968a1a..d30acebc2164 100644
>> --- a/mm/Makefile
>> +++ b/mm/Makefile
>> @@ -92,6 +92,7 @@ obj-$(CONFIG_KFENCE) += kfence/
>> obj-$(CONFIG_FAILSLAB) += failslab.o
>> obj-$(CONFIG_MEMTEST) += memtest.o
>> obj-$(CONFIG_MIGRATION) += migrate.o
>> +obj-$(CONFIG_NUMA) += memory-tiers.o
>> obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
>> obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
>> obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>> new file mode 100644
>> index 000000000000..78b311d9bde9
>> --- /dev/null
>> +++ b/mm/memory-tiers.c
>> @@ -0,0 +1,107 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +#include <linux/types.h>
>> +#include <linux/nodemask.h>
>> +#include <linux/slab.h>
>> +#include <linux/lockdep.h>
>> +#include <linux/memory-tiers.h>
>> +
>> +struct memory_tier {
>> + /* hierarchy of memory tiers */
>> + struct list_head list;
>> + /* list of all memory types part of this tier */
>> + struct list_head memory_types;
>> + /*
>> + * start value of abstract distance. memory tier maps
>> + * an abstract distance range,
>> + * adistance_start .. adistance_start + MEMTIER_CHUNK_SIZE
>> + */
>> + int adistance_start;
>> +};
>> +
>> +struct memory_dev_type {
>> + /* list of memory types that are part of same tier as this type */
>> + struct list_head tier_sibiling;
>> + /* abstract distance for this specific memory type */
>> + int adistance;
>> + /* Nodes of same abstract distance */
>> + nodemask_t nodes;
>> + struct memory_tier *memtier;
>> +};
>> +
>> +static DEFINE_MUTEX(memory_tier_lock);
>> +static LIST_HEAD(memory_tiers);
>> +static struct memory_dev_type *node_memory_types[MAX_NUMNODES];
>> +/*
>> + * For now let's have 4 memory tier below default DRAM tier.
>> + */
>> +static struct memory_dev_type default_dram_type = {
>> + .adistance = MEMTIER_ADISTANCE_DRAM,
>> + .tier_sibiling = LIST_HEAD_INIT(default_dram_type.tier_sibiling),
>> +};
>> +
>> +static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memtype)
>> +{
>> + bool found_slot = false;
>> + struct memory_tier *memtier, *new_memtier;
>> + int adistance = memtype->adistance;
>> + unsigned int memtier_adistance_chunk_size = MEMTIER_CHUNK_SIZE;
>> +
>> + lockdep_assert_held_once(&memory_tier_lock);
>> +
>> + /*
>> + * If the memtype is already part of a memory tier,
>> + * just return that.
>> + */
>> + if (memtype->memtier)
>> + return memtype->memtier;
>> +
>> + adistance = round_down(adistance, memtier_adistance_chunk_size);
>> + list_for_each_entry(memtier, &memory_tiers, list) {
>> + if (adistance == memtier->adistance_start) {
>> + memtype->memtier = memtier;
>> + list_add(&memtype->tier_sibiling, &memtier->memory_types);
>> + return memtier;
>> + } else if (adistance < memtier->adistance_start) {
>> + found_slot = true;
>> + break;
>> + }
>> + }
>> +
>> + new_memtier = kmalloc(sizeof(struct memory_tier), GFP_KERNEL);
>> + if (!new_memtier)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + new_memtier->adistance_start = adistance;
>> + INIT_LIST_HEAD(&new_memtier->list);
>> + INIT_LIST_HEAD(&new_memtier->memory_types);
>> + if (found_slot)
>> + list_add_tail(&new_memtier->list, &memtier->list);
>> + else
>> + list_add_tail(&new_memtier->list, &memory_tiers);
>> + memtype->memtier = new_memtier;
>> + list_add(&memtype->tier_sibiling, &new_memtier->memory_types);
>> + return new_memtier;
>> +}
>> +
>> +static int __init memory_tier_init(void)
>> +{
>> + int node;
>> + struct memory_tier *memtier;
>> +
>> + mutex_lock(&memory_tier_lock);
>> + /* CPU only nodes are not part of memory tiers. */
>> + default_dram_type.nodes = node_states[N_MEMORY];
>> +
>> + memtier = find_create_memory_tier(&default_dram_type);
>> + if (IS_ERR(memtier))
>> + panic("%s() failed to register memory tier: %ld\n",
>> + __func__, PTR_ERR(memtier));
>> +
>> + for_each_node_state(node, N_MEMORY)
>> + node_memory_types[node] = &default_dram_type;
>
> Although not absolutely necessary, it seems better to set
> node_memory_types[] before adding nodes to the memory type and adding
> the memory type to the memory tier.
>
>
updated.
-aneesh
Aneesh Kumar K V <[email protected]> writes:
> On 8/9/22 10:51 AM, Huang, Ying wrote:
>> "Aneesh Kumar K.V" <[email protected]> writes:
>>
>>> Also update different helpes to use NODE_DATA()->memtier. Since
>>> node specific memtier can change based on the reassignment of
>>> NUMA node to a different memory tiers, accessing NODE_DATA()->memtier
>>> needs to happen under an rcu read lock or memory_tier_lock.
>>>
>>> Signed-off-by: Aneesh Kumar K.V <[email protected]>
>>> ---
>>> include/linux/mmzone.h | 3 +++
>>> mm/memory-tiers.c | 38 ++++++++++++++++++++++++++++++++------
>>> 2 files changed, 35 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>>> index aab70355d64f..353812495a70 100644
>>> --- a/include/linux/mmzone.h
>>> +++ b/include/linux/mmzone.h
>>> @@ -928,6 +928,9 @@ typedef struct pglist_data {
>>> /* Per-node vmstats */
>>> struct per_cpu_nodestat __percpu *per_cpu_nodestats;
>>> atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
>>> +#ifdef CONFIG_NUMA
>>> + struct memory_tier __rcu *memtier;
>>> +#endif
>>> } pg_data_t;
>>>
>>> #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
>>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>>> index 02e514e87d5c..3778ac6a44a1 100644
>>> --- a/mm/memory-tiers.c
>>> +++ b/mm/memory-tiers.c
>>> @@ -5,6 +5,7 @@
>>> #include <linux/kobject.h>
>>> #include <linux/memory.h>
>>> #include <linux/random.h>
>>> +#include <linux/mmzone.h>
>>> #include <linux/memory-tiers.h>
>>>
>>> #include "internal.h"
>>> @@ -137,12 +138,18 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty
>>>
>>> static struct memory_tier *__node_get_memory_tier(int node)
>>> {
>>> - struct memory_dev_type *memtype;
>>> + pg_data_t *pgdat;
>>>
>>> - memtype = node_memory_types[node];
>>> - if (memtype && node_isset(node, memtype->nodes))
>>> - return memtype->memtier;
>>> - return NULL;
>>
>> After adding pgdat->memtier, it appears there's unnecessary to keep
>> memtype->memtier?
>>
>
> It do simplify find_create_memory_tier() where I use if (memtype->memtier)
> to check whether the memtype is already added to a memory tier. I could switch
> that to list_empty(memtype->tier_sibiling). But I felt the current one is much
> cleaner
I prefer "list_empty(memtype->tier_sibiling)". But I will let you to
decide.
Best Regards,
Huang, Ying