The changes since the last post are as follows,
- Rebased on v5.16-rc1
- Revise error processing for [1/6] (promotion counter) per Yang's comments
- Add sysctl document for [2/6] (optimize page placement)
- Reset threshold adjustment state when disable/enable tiering mode
- Reset threshold when workload transition is detected.
--
With the advent of various new memory types, some machines will have
multiple types of memory, e.g. DRAM and PMEM (persistent memory). The
memory subsystem of these machines can be called memory tiering
system, because the performance of the different types of memory are
different.
After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory
for use like normal RAM"), the PMEM could be used as the
cost-effective volatile memory in separate NUMA nodes. In a typical
memory tiering system, there are CPUs, DRAM and PMEM in each physical
NUMA node. The CPUs and the DRAM will be put in one logical node,
while the PMEM will be put in another (faked) logical node.
To optimize the system overall performance, the hot pages should be
placed in DRAM node. To do that, we need to identify the hot pages in
the PMEM node and migrate them to DRAM node via NUMA migration.
In the original NUMA balancing, there are already a set of existing
mechanisms to identify the pages recently accessed by the CPUs in a
node and migrate the pages to the node. So we can reuse these
mechanisms to build the mechanisms to optimize the page placement in
the memory tiering system. This is implemented in this patchset.
At the other hand, the cold pages should be placed in PMEM node. So,
we also need to identify the cold pages in the DRAM node and migrate
them to PMEM node.
In commit 26aa2d199d6f ("mm/migrate: demote pages during reclaim"), a
mechanism to demote the cold DRAM pages to PMEM node under memory
pressure is implemented. Based on that, the cold DRAM pages can be
demoted to PMEM node proactively to free some memory space on DRAM
node to accommodate the promoted hot PMEM pages. This is implemented
in this patchset too.
We have tested the solution with the pmbench memory accessing
benchmark with the 80:20 read/write ratio and the normal access
address distribution on a 2 socket Intel server with Optane DC
Persistent Memory Model. The test results of the base kernel and step
by step optimizations are as follows,
Throughput Promotion DRAM bandwidth
access/s MB/s MB/s
----------- ---------- --------------
Base 69263986.8 1830.2
Patch 2 135691921.4 385.6 11315.9
Patch 3 133239016.8 384.7 11065.2
Patch 4 151310868.9 197.6 11397.0
Patch 5 142311252.8 99.3 9580.8
Patch 6 149044263.9 65.5 9922.8
The whole patchset improves the benchmark score up to 115.2%. The
basic NUMA balancing based optimization solution (patch 2), the hot
page selection algorithm (patch 4), and the threshold automatic
adjustment algorithms (patch 6) improves the performance or reduce the
overhead (promotion MB/s) greatly.
Changelog:
v10:
- Rebased on v5.16-rc1
- Revise error processing for [1/6] (promotion counter) per Yang's comments
- Add sysctl document for [2/6] (optimize page placement)
- Reset threshold adjustment state when disable/enable tiering mode
- Reset threshold when workload transition is detected.
v9:
- Rebased on v5.15-rc4
- Make "add promotion counter" the first patch per Yang's comments
v8:
- Rebased on v5.15-rc1
- Make user-specified threshold take effect sooner
v7:
- Rebased on the mmots tree of 2021-07-15.
- Some minor fixes.
v6:
- Rebased on the latest page demotion patchset. (which bases on v5.11)
v5:
- Rebased on the latest page demotion patchset. (which bases on v5.10)
v4:
- Rebased on the latest page demotion patchset. (which bases on v5.9-rc6)
- Add page promotion counter.
v3:
- Move the rate limit control as late as possible per Mel Gorman's
comments.
- Revise the hot page selection implementation to store page scan time
in struct page.
- Code cleanup.
- Rebased on the latest page demotion patchset.
v2:
- Addressed comments for V1.
- Rebased on v5.5.
Best Regards,
Huang, Ying
If the NUMA balancing isn't used to optimize the page placement among
sockets but only among memory types, the hot pages in the fast memory
node couldn't be migrated (promoted) to anywhere. So it's unnecessary
to scan the pages in the fast memory node via changing their PTE/PMD
mapping to be PROT_NONE. So that the page faults could be avoided
too.
In the test, if only the memory tiering NUMA balancing mode is enabled, the
number of the NUMA balancing hint faults for the DRAM node is reduced to
almost 0 with the patch. While the benchmark score doesn't change
visibly.
Signed-off-by: "Huang, Ying" <[email protected]>
Suggested-by: Dave Hansen <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Rik van Riel <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Yang Shi <[email protected]>
Cc: Zi Yan <[email protected]>
Cc: Wei Xu <[email protected]>
Cc: osalvador <[email protected]>
Cc: Shakeel Butt <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
mm/huge_memory.c | 30 +++++++++++++++++++++---------
mm/mprotect.c | 13 ++++++++++++-
2 files changed, 33 insertions(+), 10 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index e5483347291c..cab8048eb779 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -34,6 +34,7 @@
#include <linux/oom.h>
#include <linux/numa.h>
#include <linux/page_owner.h>
+#include <linux/sched/sysctl.h>
#include <asm/tlb.h>
#include <asm/pgalloc.h>
@@ -1766,17 +1767,28 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
}
#endif
- /*
- * Avoid trapping faults against the zero page. The read-only
- * data is likely to be read-cached on the local CPU and
- * local/remote hits to the zero page are not interesting.
- */
- if (prot_numa && is_huge_zero_pmd(*pmd))
- goto unlock;
+ if (prot_numa) {
+ struct page *page;
+ /*
+ * Avoid trapping faults against the zero page. The read-only
+ * data is likely to be read-cached on the local CPU and
+ * local/remote hits to the zero page are not interesting.
+ */
+ if (is_huge_zero_pmd(*pmd))
+ goto unlock;
- if (prot_numa && pmd_protnone(*pmd))
- goto unlock;
+ if (pmd_protnone(*pmd))
+ goto unlock;
+ page = pmd_page(*pmd);
+ /*
+ * Skip scanning top tier node if normal numa
+ * balancing is disabled
+ */
+ if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
+ node_is_toptier(page_to_nid(page)))
+ goto unlock;
+ }
/*
* In case prot_numa, we are under mmap_read_lock(mm). It's critical
* to not clear pmd intermittently to avoid race with MADV_DONTNEED
diff --git a/mm/mprotect.c b/mm/mprotect.c
index e552f5e0ccbd..ddc24ca52b12 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -29,6 +29,7 @@
#include <linux/uaccess.h>
#include <linux/mm_inline.h>
#include <linux/pgtable.h>
+#include <linux/sched/sysctl.h>
#include <asm/cacheflush.h>
#include <asm/mmu_context.h>
#include <asm/tlbflush.h>
@@ -83,6 +84,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
*/
if (prot_numa) {
struct page *page;
+ int nid;
/* Avoid TLB flush if possible */
if (pte_protnone(oldpte))
@@ -109,7 +111,16 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
* Don't mess with PTEs if page is already on the node
* a single-threaded process is running on.
*/
- if (target_node == page_to_nid(page))
+ nid = page_to_nid(page);
+ if (target_node == nid)
+ continue;
+
+ /*
+ * Skip scanning top tier node if normal numa
+ * balancing is disabled
+ */
+ if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
+ node_is_toptier(nid))
continue;
}
--
2.30.2
With the advent of various new memory types, some machines will have
multiple types of memory, e.g. DRAM and PMEM (persistent memory). The
memory subsystem of these machines can be called memory tiering
system, because the performance of the different types of memory are
usually different.
In such system, because of the memory accessing pattern changing etc,
some pages in the slow memory may become hot globally. So in this
patch, the NUMA balancing mechanism is enhanced to optimize the page
placement among the different memory types according to hot/cold
dynamically.
In a typical memory tiering system, there are CPUs, fast memory and
slow memory in each physical NUMA node. The CPUs and the fast memory
will be put in one logical node (called fast memory node), while the
slow memory will be put in another (faked) logical node (called slow
memory node). That is, the fast memory is regarded as local while the
slow memory is regarded as remote. So it's possible for the recently
accessed pages in the slow memory node to be promoted to the fast
memory node via the existing NUMA balancing mechanism.
The original NUMA balancing mechanism will stop to migrate pages if the free
memory of the target node will become below the high watermark. This
is a reasonable policy if there's only one memory type. But this
makes the original NUMA balancing mechanism almost not work to optimize page
placement among different memory types. Details are as follows.
It's the common cases that the working-set size of the workload is
larger than the size of the fast memory nodes. Otherwise, it's
unnecessary to use the slow memory at all. So in the common cases,
there are almost always no enough free pages in the fast memory nodes,
so that the globally hot pages in the slow memory node cannot be
promoted to the fast memory node. To solve the issue, we have 2
choices as follows,
a. Ignore the free pages watermark checking when promoting hot pages
from the slow memory node to the fast memory node. This will
create some memory pressure in the fast memory node, thus trigger
the memory reclaiming. So that, the cold pages in the fast memory
node will be demoted to the slow memory node.
b. Make kswapd of the fast memory node to reclaim pages until the free
pages are a little more (about 10MB) than the high watermark. Then,
if the free pages of the fast memory node reaches high watermark, and
some hot pages need to be promoted, kswapd of the fast memory node
will be waken up to demote some cold pages in the fast memory node to
the slow memory node. This will free some extra space in the fast
memory node, so the hot pages in the slow memory node can be
promoted to the fast memory node.
The choice "a" will create the memory pressure in the fast memory
node. If the memory pressure of the workload is high, the memory
pressure may become so high that the memory allocation latency of the
workload is influenced, e.g. the direct reclaiming may be triggered.
The choice "b" works much better at this aspect. If the memory
pressure of the workload is high, the hot pages promotion will stop
earlier because its allocation watermark is higher than that of the
normal memory allocation. So in this patch, choice "b" is
implemented.
In addition to the original page placement optimization among sockets,
the NUMA balancing mechanism is extended to be used to optimize page
placement according to hot/cold among different memory types. So the
sysctl user space interface (numa_balancing) is extended in a backward
compatible way as follow, so that the users can enable/disable these
functionality individually.
The sysctl is converted from a Boolean value to a bits field. The
definition of the flags is,
- 0x0: NUMA_BALANCING_DISABLED
- 0x1: NUMA_BALANCING_NORMAL
- 0x2: NUMA_BALANCING_MEMORY_TIERING
Signed-off-by: "Huang, Ying" <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Rik van Riel <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Yang Shi <[email protected]>
Cc: Zi Yan <[email protected]>
Cc: Wei Xu <[email protected]>
Cc: osalvador <[email protected]>
Cc: Shakeel Butt <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
Documentation/admin-guide/sysctl/kernel.rst | 29 ++++++++++++++-------
include/linux/sched/sysctl.h | 10 +++++++
kernel/sched/core.c | 21 ++++++++++++---
kernel/sysctl.c | 3 ++-
mm/migrate.c | 19 ++++++++++++--
mm/vmscan.c | 16 ++++++++++++
6 files changed, 82 insertions(+), 16 deletions(-)
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 426162009ce9..1974a559c10b 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst).
numa_balancing
==============
-Enables/disables automatic page fault based NUMA memory
-balancing. Memory is moved automatically to nodes
-that access it often.
+Enables/disables and configure automatic page fault based NUMA memory
+balancing. Memory is moved automatically to nodes that access it
+often. The value to set can be the result to OR the following,
-Enables/disables automatic NUMA memory balancing. On NUMA machines, there
-is a performance penalty if remote memory is accessed by a CPU. When this
-feature is enabled the kernel samples what task thread is accessing memory
-by periodically unmapping pages and later trapping a page fault. At the
-time of the page fault, it is determined if the data being accessed should
-be migrated to a local memory node.
+= =================================
+0x0 NUMA_BALANCING_DISABLED
+0x1 NUMA_BALANCING_NORMAL
+0x2 NUMA_BALANCING_MEMORY_TIERING
+= =================================
+
+Or NUMA_BALANCING_NORMAL to optimize page placement among different
+NUMA nodes to reduce remote accessing. On NUMA machines, there is a
+performance penalty if remote memory is accessed by a CPU. When this
+feature is enabled the kernel samples what task thread is accessing
+memory by periodically unmapping pages and later trapping a page
+fault. At the time of the page fault, it is determined if the data
+being accessed should be migrated to a local memory node.
The unmapping of pages and trapping faults incur additional overhead that
ideally is offset by improved memory locality but there is no universal
@@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms,
numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms,
numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls.
+Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among
+different types of memory (represented as different NUMA nodes) to
+place the hot pages in the fast memory. This is implemented based on
+unmapping and page fault too.
numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb
===============================================================================================================================
diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h
index 304f431178fd..bc54c1d75d6d 100644
--- a/include/linux/sched/sysctl.h
+++ b/include/linux/sched/sysctl.h
@@ -35,6 +35,16 @@ enum sched_tunable_scaling {
SCHED_TUNABLESCALING_END,
};
+#define NUMA_BALANCING_DISABLED 0x0
+#define NUMA_BALANCING_NORMAL 0x1
+#define NUMA_BALANCING_MEMORY_TIERING 0x2
+
+#ifdef CONFIG_NUMA_BALANCING
+extern int sysctl_numa_balancing_mode;
+#else
+#define sysctl_numa_balancing_mode 0
+#endif
+
/*
* control realtime throttling:
*
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 3c9b0fda64ac..5dcabc98432f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4265,7 +4265,9 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing);
#ifdef CONFIG_NUMA_BALANCING
-void set_numabalancing_state(bool enabled)
+int sysctl_numa_balancing_mode;
+
+static void __set_numabalancing_state(bool enabled)
{
if (enabled)
static_branch_enable(&sched_numa_balancing);
@@ -4273,13 +4275,22 @@ void set_numabalancing_state(bool enabled)
static_branch_disable(&sched_numa_balancing);
}
+void set_numabalancing_state(bool enabled)
+{
+ if (enabled)
+ sysctl_numa_balancing_mode = NUMA_BALANCING_NORMAL;
+ else
+ sysctl_numa_balancing_mode = NUMA_BALANCING_DISABLED;
+ __set_numabalancing_state(enabled);
+}
+
#ifdef CONFIG_PROC_SYSCTL
int sysctl_numa_balancing(struct ctl_table *table, int write,
void *buffer, size_t *lenp, loff_t *ppos)
{
struct ctl_table t;
int err;
- int state = static_branch_likely(&sched_numa_balancing);
+ int state = sysctl_numa_balancing_mode;
if (write && !capable(CAP_SYS_ADMIN))
return -EPERM;
@@ -4289,8 +4300,10 @@ int sysctl_numa_balancing(struct ctl_table *table, int write,
err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos);
if (err < 0)
return err;
- if (write)
- set_numabalancing_state(state);
+ if (write) {
+ sysctl_numa_balancing_mode = state;
+ __set_numabalancing_state(state);
+ }
return err;
}
#endif
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 083be6af29d7..a1be94ea80ba 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -115,6 +115,7 @@ static int sixty = 60;
static int __maybe_unused neg_one = -1;
static int __maybe_unused two = 2;
+static int __maybe_unused three = 3;
static int __maybe_unused four = 4;
static unsigned long zero_ul;
static unsigned long one_ul = 1;
@@ -1808,7 +1809,7 @@ static struct ctl_table kern_table[] = {
.mode = 0644,
.proc_handler = sysctl_numa_balancing,
.extra1 = SYSCTL_ZERO,
- .extra2 = SYSCTL_ONE,
+ .extra2 = &three,
},
#endif /* CONFIG_NUMA_BALANCING */
{
diff --git a/mm/migrate.c b/mm/migrate.c
index b7c27abb0e5c..286c84c014dd 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -50,6 +50,7 @@
#include <linux/ptrace.h>
#include <linux/oom.h>
#include <linux/memory.h>
+#include <linux/sched/sysctl.h>
#include <asm/tlbflush.h>
@@ -2103,16 +2104,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)
{
int page_lru;
int nr_pages = thp_nr_pages(page);
+ int order = compound_order(page);
- VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page);
+ VM_BUG_ON_PAGE(order && !PageTransHuge(page), page);
/* Do not migrate THP mapped by multiple processes */
if (PageTransHuge(page) && total_mapcount(page) > 1)
return 0;
/* Avoid migrating to a node that is nearly full */
- if (!migrate_balanced_pgdat(pgdat, nr_pages))
+ if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
+ int z;
+
+ if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) ||
+ !numa_demotion_enabled)
+ return 0;
+ if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE)
+ return 0;
+ for (z = pgdat->nr_zones - 1; z >= 0; z--) {
+ if (populated_zone(pgdat->node_zones + z))
+ break;
+ }
+ wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE);
return 0;
+ }
if (isolate_lru_page(page))
return 0;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index fb9584641ac7..8ec955404bd1 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -56,6 +56,7 @@
#include <linux/swapops.h>
#include <linux/balloon_compaction.h>
+#include <linux/sched/sysctl.h>
#include "internal.h"
@@ -3908,6 +3909,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx)
return false;
}
+/*
+ * Keep the free pages on fast memory node a little more than the high
+ * watermark to accommodate the promoted pages.
+ */
+#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT)
+
/*
* Returns true if there is an eligible zone balanced for the request order
* and highest_zoneidx
@@ -3929,6 +3936,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
continue;
mark = high_wmark_pages(zone);
+ if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
+ numa_demotion_enabled &&
+ next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) {
+ unsigned long promote_mark;
+
+ promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK,
+ pgdat->node_present_pages >> 6);
+ mark += promote_mark;
+ }
if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx))
return true;
}
--
2.30.2
On 15 Nov 2021, at 20:35, Huang Ying wrote:
> With the advent of various new memory types, some machines will have
> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The
> memory subsystem of these machines can be called memory tiering
> system, because the performance of the different types of memory are
> usually different.
>
> In such system, because of the memory accessing pattern changing etc,
> some pages in the slow memory may become hot globally. So in this
> patch, the NUMA balancing mechanism is enhanced to optimize the page
> placement among the different memory types according to hot/cold
> dynamically.
>
> In a typical memory tiering system, there are CPUs, fast memory and
> slow memory in each physical NUMA node. The CPUs and the fast memory
> will be put in one logical node (called fast memory node), while the
> slow memory will be put in another (faked) logical node (called slow
> memory node). That is, the fast memory is regarded as local while the
> slow memory is regarded as remote. So it's possible for the recently
> accessed pages in the slow memory node to be promoted to the fast
> memory node via the existing NUMA balancing mechanism.
>
> The original NUMA balancing mechanism will stop to migrate pages if the free
> memory of the target node will become below the high watermark. This
> is a reasonable policy if there's only one memory type. But this
> makes the original NUMA balancing mechanism almost not work to optimize page
> placement among different memory types. Details are as follows.
>
> It's the common cases that the working-set size of the workload is
> larger than the size of the fast memory nodes. Otherwise, it's
> unnecessary to use the slow memory at all. So in the common cases,
> there are almost always no enough free pages in the fast memory nodes,
> so that the globally hot pages in the slow memory node cannot be
> promoted to the fast memory node. To solve the issue, we have 2
> choices as follows,
>
> a. Ignore the free pages watermark checking when promoting hot pages
> from the slow memory node to the fast memory node. This will
> create some memory pressure in the fast memory node, thus trigger
> the memory reclaiming. So that, the cold pages in the fast memory
> node will be demoted to the slow memory node.
>
> b. Make kswapd of the fast memory node to reclaim pages until the free
> pages are a little more (about 10MB) than the high watermark. Then,
> if the free pages of the fast memory node reaches high watermark, and
> some hot pages need to be promoted, kswapd of the fast memory node
> will be waken up to demote some cold pages in the fast memory node to
> the slow memory node. This will free some extra space in the fast
> memory node, so the hot pages in the slow memory node can be
> promoted to the fast memory node.
Why 10MB? Is 10MB big enough to avoid creating memory pressure on fast
memory? This number seems pretty ad-hoc and may only work well on your
test machine.
In theory, this extra free memory space should be related to page promotion
throughput and kswapd demotion throughput, right? Patch 5 allows user
to adjust page promotion throughput, NUMA_BALANCING_PROMOTE_WATERMARK
at least can be something like X * numa_balancing_rate_limit_mbps.
Also, is there any way of measuring kswapd demotion throughput at boot
time? So we can take it into account too. Does this make sense?
>
> The choice "a" will create the memory pressure in the fast memory
> node. If the memory pressure of the workload is high, the memory
> pressure may become so high that the memory allocation latency of the
> workload is influenced, e.g. the direct reclaiming may be triggered.
>
> The choice "b" works much better at this aspect. If the memory
> pressure of the workload is high, the hot pages promotion will stop
> earlier because its allocation watermark is higher than that of the
> normal memory allocation. So in this patch, choice "b" is
> implemented.
>
> In addition to the original page placement optimization among sockets,
> the NUMA balancing mechanism is extended to be used to optimize page
> placement according to hot/cold among different memory types. So the
> sysctl user space interface (numa_balancing) is extended in a backward
> compatible way as follow, so that the users can enable/disable these
> functionality individually.
>
> The sysctl is converted from a Boolean value to a bits field. The
> definition of the flags is,
>
> - 0x0: NUMA_BALANCING_DISABLED
> - 0x1: NUMA_BALANCING_NORMAL
> - 0x2: NUMA_BALANCING_MEMORY_TIERING
>
> Signed-off-by: "Huang, Ying" <[email protected]>
> Cc: Andrew Morton <[email protected]>
> Cc: Michal Hocko <[email protected]>
> Cc: Rik van Riel <[email protected]>
> Cc: Mel Gorman <[email protected]>
> Cc: Peter Zijlstra <[email protected]>
> Cc: Dave Hansen <[email protected]>
> Cc: Yang Shi <[email protected]>
> Cc: Zi Yan <[email protected]>
> Cc: Wei Xu <[email protected]>
> Cc: osalvador <[email protected]>
> Cc: Shakeel Butt <[email protected]>
> Cc: [email protected]
> Cc: [email protected]
> ---
> Documentation/admin-guide/sysctl/kernel.rst | 29 ++++++++++++++-------
> include/linux/sched/sysctl.h | 10 +++++++
> kernel/sched/core.c | 21 ++++++++++++---
> kernel/sysctl.c | 3 ++-
> mm/migrate.c | 19 ++++++++++++--
> mm/vmscan.c | 16 ++++++++++++
> 6 files changed, 82 insertions(+), 16 deletions(-)
>
> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index 426162009ce9..1974a559c10b 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst
> @@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst).
> numa_balancing
> ==============
>
> -Enables/disables automatic page fault based NUMA memory
> -balancing. Memory is moved automatically to nodes
> -that access it often.
> +Enables/disables and configure automatic page fault based NUMA memory
> +balancing. Memory is moved automatically to nodes that access it
> +often. The value to set can be the result to OR the following,
>
> -Enables/disables automatic NUMA memory balancing. On NUMA machines, there
> -is a performance penalty if remote memory is accessed by a CPU. When this
> -feature is enabled the kernel samples what task thread is accessing memory
> -by periodically unmapping pages and later trapping a page fault. At the
> -time of the page fault, it is determined if the data being accessed should
> -be migrated to a local memory node.
> += =================================
> +0x0 NUMA_BALANCING_DISABLED
> +0x1 NUMA_BALANCING_NORMAL
> +0x2 NUMA_BALANCING_MEMORY_TIERING
> += =================================
> +
> +Or NUMA_BALANCING_NORMAL to optimize page placement among different
> +NUMA nodes to reduce remote accessing. On NUMA machines, there is a
> +performance penalty if remote memory is accessed by a CPU. When this
> +feature is enabled the kernel samples what task thread is accessing
> +memory by periodically unmapping pages and later trapping a page
> +fault. At the time of the page fault, it is determined if the data
> +being accessed should be migrated to a local memory node.
>
> The unmapping of pages and trapping faults incur additional overhead that
> ideally is offset by improved memory locality but there is no universal
> @@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms,
> numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms,
> numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls.
>
> +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among
> +different types of memory (represented as different NUMA nodes) to
> +place the hot pages in the fast memory. This is implemented based on
> +unmapping and page fault too.
>
> numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb
> ===============================================================================================================================
> diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h
> index 304f431178fd..bc54c1d75d6d 100644
> --- a/include/linux/sched/sysctl.h
> +++ b/include/linux/sched/sysctl.h
> @@ -35,6 +35,16 @@ enum sched_tunable_scaling {
> SCHED_TUNABLESCALING_END,
> };
>
> +#define NUMA_BALANCING_DISABLED 0x0
> +#define NUMA_BALANCING_NORMAL 0x1
> +#define NUMA_BALANCING_MEMORY_TIERING 0x2
> +
> +#ifdef CONFIG_NUMA_BALANCING
> +extern int sysctl_numa_balancing_mode;
> +#else
> +#define sysctl_numa_balancing_mode 0
> +#endif
> +
> /*
> * control realtime throttling:
> *
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 3c9b0fda64ac..5dcabc98432f 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4265,7 +4265,9 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing);
>
> #ifdef CONFIG_NUMA_BALANCING
>
> -void set_numabalancing_state(bool enabled)
> +int sysctl_numa_balancing_mode;
> +
> +static void __set_numabalancing_state(bool enabled)
> {
> if (enabled)
> static_branch_enable(&sched_numa_balancing);
> @@ -4273,13 +4275,22 @@ void set_numabalancing_state(bool enabled)
> static_branch_disable(&sched_numa_balancing);
> }
>
> +void set_numabalancing_state(bool enabled)
> +{
> + if (enabled)
> + sysctl_numa_balancing_mode = NUMA_BALANCING_NORMAL;
> + else
> + sysctl_numa_balancing_mode = NUMA_BALANCING_DISABLED;
> + __set_numabalancing_state(enabled);
> +}
> +
> #ifdef CONFIG_PROC_SYSCTL
> int sysctl_numa_balancing(struct ctl_table *table, int write,
> void *buffer, size_t *lenp, loff_t *ppos)
> {
> struct ctl_table t;
> int err;
> - int state = static_branch_likely(&sched_numa_balancing);
> + int state = sysctl_numa_balancing_mode;
>
> if (write && !capable(CAP_SYS_ADMIN))
> return -EPERM;
> @@ -4289,8 +4300,10 @@ int sysctl_numa_balancing(struct ctl_table *table, int write,
> err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos);
> if (err < 0)
> return err;
> - if (write)
> - set_numabalancing_state(state);
> + if (write) {
> + sysctl_numa_balancing_mode = state;
> + __set_numabalancing_state(state);
> + }
> return err;
> }
> #endif
> diff --git a/kernel/sysctl.c b/kernel/sysctl.c
> index 083be6af29d7..a1be94ea80ba 100644
> --- a/kernel/sysctl.c
> +++ b/kernel/sysctl.c
> @@ -115,6 +115,7 @@ static int sixty = 60;
>
> static int __maybe_unused neg_one = -1;
> static int __maybe_unused two = 2;
> +static int __maybe_unused three = 3;
> static int __maybe_unused four = 4;
> static unsigned long zero_ul;
> static unsigned long one_ul = 1;
> @@ -1808,7 +1809,7 @@ static struct ctl_table kern_table[] = {
> .mode = 0644,
> .proc_handler = sysctl_numa_balancing,
> .extra1 = SYSCTL_ZERO,
> - .extra2 = SYSCTL_ONE,
> + .extra2 = &three,
> },
> #endif /* CONFIG_NUMA_BALANCING */
> {
> diff --git a/mm/migrate.c b/mm/migrate.c
> index b7c27abb0e5c..286c84c014dd 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -50,6 +50,7 @@
> #include <linux/ptrace.h>
> #include <linux/oom.h>
> #include <linux/memory.h>
> +#include <linux/sched/sysctl.h>
>
> #include <asm/tlbflush.h>
>
> @@ -2103,16 +2104,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)
> {
> int page_lru;
> int nr_pages = thp_nr_pages(page);
> + int order = compound_order(page);
>
> - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page);
> + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page);
>
> /* Do not migrate THP mapped by multiple processes */
> if (PageTransHuge(page) && total_mapcount(page) > 1)
> return 0;
>
> /* Avoid migrating to a node that is nearly full */
> - if (!migrate_balanced_pgdat(pgdat, nr_pages))
> + if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
> + int z;
> +
> + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) ||
> + !numa_demotion_enabled)
> + return 0;
> + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE)
> + return 0;
> + for (z = pgdat->nr_zones - 1; z >= 0; z--) {
> + if (populated_zone(pgdat->node_zones + z))
> + break;
> + }
> + wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE);
> return 0;
> + }
>
> if (isolate_lru_page(page))
> return 0;
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index fb9584641ac7..8ec955404bd1 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -56,6 +56,7 @@
>
> #include <linux/swapops.h>
> #include <linux/balloon_compaction.h>
> +#include <linux/sched/sysctl.h>
>
> #include "internal.h"
>
> @@ -3908,6 +3909,12 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx)
> return false;
> }
>
> +/*
> + * Keep the free pages on fast memory node a little more than the high
> + * watermark to accommodate the promoted pages.
> + */
> +#define NUMA_BALANCING_PROMOTE_WATERMARK (10UL * 1024 * 1024 >> PAGE_SHIFT)
> +
> /*
> * Returns true if there is an eligible zone balanced for the request order
> * and highest_zoneidx
> @@ -3929,6 +3936,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
> continue;
>
> mark = high_wmark_pages(zone);
> + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
> + numa_demotion_enabled &&
> + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) {
> + unsigned long promote_mark;
> +
> + promote_mark = min(NUMA_BALANCING_PROMOTE_WATERMARK,
> + pgdat->node_present_pages >> 6);
> + mark += promote_mark;
> + }
> if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx))
> return true;
> }
> --
> 2.30.2
--
Best Regards,
Yan, Zi
Hi, Zi,
Thanks for review!
Zi Yan <[email protected]> writes:
> On 15 Nov 2021, at 20:35, Huang Ying wrote:
>
>> With the advent of various new memory types, some machines will have
>> multiple types of memory, e.g. DRAM and PMEM (persistent memory). The
>> memory subsystem of these machines can be called memory tiering
>> system, because the performance of the different types of memory are
>> usually different.
>>
>> In such system, because of the memory accessing pattern changing etc,
>> some pages in the slow memory may become hot globally. So in this
>> patch, the NUMA balancing mechanism is enhanced to optimize the page
>> placement among the different memory types according to hot/cold
>> dynamically.
>>
>> In a typical memory tiering system, there are CPUs, fast memory and
>> slow memory in each physical NUMA node. The CPUs and the fast memory
>> will be put in one logical node (called fast memory node), while the
>> slow memory will be put in another (faked) logical node (called slow
>> memory node). That is, the fast memory is regarded as local while the
>> slow memory is regarded as remote. So it's possible for the recently
>> accessed pages in the slow memory node to be promoted to the fast
>> memory node via the existing NUMA balancing mechanism.
>>
>> The original NUMA balancing mechanism will stop to migrate pages if the free
>> memory of the target node will become below the high watermark. This
>> is a reasonable policy if there's only one memory type. But this
>> makes the original NUMA balancing mechanism almost not work to optimize page
>> placement among different memory types. Details are as follows.
>>
>> It's the common cases that the working-set size of the workload is
>> larger than the size of the fast memory nodes. Otherwise, it's
>> unnecessary to use the slow memory at all. So in the common cases,
>> there are almost always no enough free pages in the fast memory nodes,
>> so that the globally hot pages in the slow memory node cannot be
>> promoted to the fast memory node. To solve the issue, we have 2
>> choices as follows,
>>
>> a. Ignore the free pages watermark checking when promoting hot pages
>> from the slow memory node to the fast memory node. This will
>> create some memory pressure in the fast memory node, thus trigger
>> the memory reclaiming. So that, the cold pages in the fast memory
>> node will be demoted to the slow memory node.
>>
>> b. Make kswapd of the fast memory node to reclaim pages until the free
>> pages are a little more (about 10MB) than the high watermark. Then,
>> if the free pages of the fast memory node reaches high watermark, and
>> some hot pages need to be promoted, kswapd of the fast memory node
>> will be waken up to demote some cold pages in the fast memory node to
>> the slow memory node. This will free some extra space in the fast
>> memory node, so the hot pages in the slow memory node can be
>> promoted to the fast memory node.
>
> Why 10MB? Is 10MB big enough to avoid creating memory pressure on fast
> memory? This number seems pretty ad-hoc and may only work well on your
> test machine.
>
> In theory, this extra free memory space should be related to page promotion
> throughput and kswapd demotion throughput, right? Patch 5 allows user
> to adjust page promotion throughput, NUMA_BALANCING_PROMOTE_WATERMARK
> at least can be something like X * numa_balancing_rate_limit_mbps.
> Also, is there any way of measuring kswapd demotion throughput at boot
> time? So we can take it into account too. Does this make sense?
Yes. 10MB is just a ad-hoc number that happens work well on my test
machine. And I am glad to discuss the proper value.
- The effect of the patchset will keep the size of the free memory in
the DRAM node between HIGH_WATERMARK and HIGH_WATERMARK +
PROMOTE_WATERMARK in normal cases. So PROMOTE_WATERMARK should be
as small as possible to take full advantage of the valuable DRAM.
- The PROMOTE_WATERMARK should be larger than the max per-CPU counter
error. So that migrate_balanced_pgdat() for promotion and
pgdat_balanced() for demotion can cooperate.
- The PROMOTE_WATERMARK should be large enough to batch the demotion.
- If the promotion speed is slower than the demotion speed, the larger
the PROMOTE_WATERMARK, the longer the sleep time of kswapd. If the
promotion speed is faster than the demotion speed, kswapd will keep
running until promotion is rate limited at least.
So, if we set PROMOTE_WATERMARK as N * numa_balancing_rate_limit_mbps,
then kswapd will sleep about N seconds after the number of the free
pages reaches PROMOTE_WATERMARK. Is that expected?
All in all, I don't think some ad-hoc number such as 10MB is good
enough. And the solution adopted in this patch should be as simple as
possible. This will provide a baseline, so that we can improve it later
with more sophisticated solution with the numbers. Do you agree?
Best Regards,
Huang, Ying
>>
>> The choice "a" will create the memory pressure in the fast memory
>> node. If the memory pressure of the workload is high, the memory
>> pressure may become so high that the memory allocation latency of the
>> workload is influenced, e.g. the direct reclaiming may be triggered.
>>
>> The choice "b" works much better at this aspect. If the memory
>> pressure of the workload is high, the hot pages promotion will stop
>> earlier because its allocation watermark is higher than that of the
>> normal memory allocation. So in this patch, choice "b" is
>> implemented.
>>
>> In addition to the original page placement optimization among sockets,
>> the NUMA balancing mechanism is extended to be used to optimize page
>> placement according to hot/cold among different memory types. So the
>> sysctl user space interface (numa_balancing) is extended in a backward
>> compatible way as follow, so that the users can enable/disable these
>> functionality individually.
>>
>> The sysctl is converted from a Boolean value to a bits field. The
>> definition of the flags is,
>>
>> - 0x0: NUMA_BALANCING_DISABLED
>> - 0x1: NUMA_BALANCING_NORMAL
>> - 0x2: NUMA_BALANCING_MEMORY_TIERING
>>
>> Signed-off-by: "Huang, Ying" <[email protected]>
>> Cc: Andrew Morton <[email protected]>
>> Cc: Michal Hocko <[email protected]>
>> Cc: Rik van Riel <[email protected]>
>> Cc: Mel Gorman <[email protected]>
>> Cc: Peter Zijlstra <[email protected]>
>> Cc: Dave Hansen <[email protected]>
>> Cc: Yang Shi <[email protected]>
>> Cc: Zi Yan <[email protected]>
>> Cc: Wei Xu <[email protected]>
>> Cc: osalvador <[email protected]>
>> Cc: Shakeel Butt <[email protected]>
>> Cc: [email protected]
>> Cc: [email protected]
[snip]
Best Regards,
Huang, Ying