Changes from RFC
================
Compared to the RFC
(https://lore.kernel.org/damon/[email protected]/), this
version of the patchset contains below changes.
- Use more self-explaining DAMOS action names
- Put more evaluation results
- Introduce a static kernel module for easy use of conservatively-tuned
DAMON-based proactive LRU-lists sorting
Introduction
============
In short, this patchset 1) extends DAMON-based Operation Schemes (DAMOS) for
low overhead data access pattern based LRU-lists sorting, and 2) implements a
static kernel module for easy use of conservatively-tuned version of that using
the extended DAMOS capability.
Background
----------
As page-granularity access checking overhead could be significant on huge
systems, LRU lists are normally not proactively sorted but partially and
reactively sorted for special events including specific user requests, system
calls and memory pressure. As a result, LRU lists are sometimes not so
perfectly prepared to be used as a trustworthy access pattern source for some
situations including reclamation target pages selection under sudden memory
pressure.
DAMON-based Proactive LRU-lists Sorting
---------------------------------------
Because DAMON can identify access patterns of best-effort accuracy while
inducing only user-specified range of overhead, using DAMON for Proactive
LRU-lists Sorting (PLRUS) could be helpful for this situation. The idea is
quite simple. Find hot pages and cold pages using DAMON, and prioritize hot
pages while deprioritizing cold pages on their LRU-lists.
This patchset extends DAMON to support such schemes by introducing a couple of
new DAMOS actions for prioritizing and deprioritizing memory regions of
specific access patterns on their LRU-lists. In detail, this patchset simply
uses 'mark_page_accessed()' and 'deactivate_page()' functions for
prioritization and deprioritization of pages on their LRU lists, respectively.
To make the scheme easy to use without complex tuning for common situations,
this patchset further implements a static kernel module called 'DAMON_LRU_SORT'
using the extended DAMOS functionality. It proactively sorts LRU-lists using
DAMON with conservatively chosen default hotness/coldness thresholds and small
CPU usage quota limit. That is, the module under its default parameters will
make no harm for common situation but provide some level of benefit for systems
having clear hot/cold access pattern under only memory pressure while consuming
only limited small portion of CPU time.
Related Works
-------------
Proactive reclamation is well known to be helpful for reducing non-optimal
reclamation target selection caused performance drops. However, proactive
reclamation is not a best option for some cases, because it could incur
additional I/O. For an example, it could be prohitive for systems using
storage devices that total number of writes is limited, or cloud block storages
that charges every I/O.
Some proactive reclamation approaches[1,2] induce a level of memory pressure
using memcg files or swappiness while monitoring PSI. As reclamation target
selection is still relying on the original LRU-lists mechanism, using
DAMON-based proactive reclamation before inducing the proactive reclamation
could allow more memory saving with same level of performance overhead, or less
performance overhead with same level of memory saving.
[1] https://blogs.oracle.com/linux/post/anticipating-your-memory-needs
[2] https://www.pdl.cmu.edu/ftp/NVM/tmo_asplos22.pdf
Evaluation
==========
In short, PLRUS achieves 10% memory PSI (some) reduction, 14% major page faults
reduction, and 3.74% speedup under memory pressure.
Setup
-----
To show the effect of PLRUS, I run PARSEC3 and SPLASH-2X benchmarks under below
variant systems and measure a few metrics including the runtime of each
workload, number of system-wide major page faults, and system-wide memory PSI
(some).
- orig: v5.18-rc4 based mm-unstable kernel + this patchset, but no DAMON scheme
applied.
- mprs: Same to 'orig' but artificial memory pressure is induced.
- plrus: Same to 'mprs' but a radically tuned PLRUS scheme is applied to the
entire physical address space of the system.
For the artificial memory pressure, I set 'memory.limit_in_bytes' to 75% of the
running workload's peak RSS, wait 1 second, remove the pressure by setting it
to 200% of the peak RSS, wait 10 seconds, and repeat the procedure until the
workload finishes[1]. I use zram based swap device. The tests are
automated[2].
[1] https://github.com/awslabs/damon-tests/blob/next/perf/runners/back/0009_memcg_pressure.sh
[2] https://github.com/awslabs/damon-tests/blob/next/perf/full_once_config.sh
Radically Tuned PLRUS
---------------------
To show effect of PLRUS on the PARSEC3/SPLASH-2X workloads which runs for no
long time, we use radically tuned version of PLRUS. The version asks DAMON to
do the proactive LRU-lists sorting as below.
1. Find any memory regions shown some accesses (approximately >=20 accesses per
100 sampling) and prioritize pages of the regions on their LRU lists using
up to 2% CPU time. Under the CPU time limit, prioritize regions having
higher access frequency and kept the access frequency longer first.
2. Find any memory regions shown no access for at least >=5 seconds and
deprioritize pages of the rgions on their LRU lists using up to 2% CPU time.
Under the CPU time limit, deprioritize regions that not accessed for longer
time first.
Results
-------
I repeat the tests 25 times and calculate average of the measured numbers. The
results are as below:
metric orig mprs plrus plrus/mprs
runtime_seconds 190.06 292.83 281.87 0.96
pgmajfaults 852.55 8769420.00 7525040.00 0.86
memory_psi_some_us 106911.00 6943420.00 6220920.00 0.90
The first row is for legend. The first cell shows the metric that the
following cells of the row shows. Second, third, and fourth cells show the
metrics under the configs shown at the first row of the cell, and the fifth
cell shows the metric under 'plrus' divided by the metric under 'mprs'. Second
row shows the averaged runtime of the workloads in seconds. Third row shows
the number of system-wide major page faults while the test was ongoing. Fourth
row shows the system-wide memory pressure stall for some processes in
microseconds while the test was ongoing.
In short, PLRUS achieves 10% memory PSI (some) reduction, 14% major page faults
reduction, and 3.74% speedup under memory pressure. We also confirmed the CPU
usage of kdamond was 2.61% of single CPU, which is below 4% as expected.
Sequence of Patches
===================
The first and second patch cleans up DAMON debugfs interface and DAMOS_PAGEOUT
handling code of physical address space monitoring operations implementation
for easier extension of the code.
The thrid and fourth patches implement a new DAMOS action called 'lru_prio',
which prioritizes pages under memory regions which have a user-specified access
pattern, and document it, respectively. The fifth and sixth patches implement
yet another new DAMOS action called 'lru_deprio', which deprioritizes pages
under memory regions which have a user-specified access pattern, and document
it, respectively.
The seventh patch implements a static kernel module called 'damon_lru_sort',
which utilizes the DAMON-based proactive LRU-lists sorting under conservatively
chosen default parameter. Finally, the eighth patch documents
'damon_lru_sort'.
SeongJae Park (8):
mm/damon/dbgfs: add and use mappings between 'schemes' action inputs
and 'damos_action' values
mm/damon/paddr: use a separate function for 'DAMOS_PAGEOUT' handling
mm/damon/schemes: add 'LRU_PRIO' DAMOS action
Docs/admin-guide/damon/sysfs: document 'LRU_PRIO' scheme action
mm/damon/schemes: add 'LRU_DEPRIO' action
Docs/admin-guide/damon/sysfs: document 'LRU_DEPRIO' scheme action
mm/damon: introduce DAMON-based LRU-lists Sorting
Docs/admin-guide/damon: add a document for DAMON_LRU_SORT
Documentation/admin-guide/mm/damon/index.rst | 1 +
.../admin-guide/mm/damon/lru_sort.rst | 294 ++++++++++
Documentation/admin-guide/mm/damon/usage.rst | 2 +
include/linux/damon.h | 4 +
mm/damon/Kconfig | 8 +
mm/damon/Makefile | 1 +
mm/damon/dbgfs.c | 64 +-
mm/damon/lru_sort.c | 546 ++++++++++++++++++
mm/damon/ops-common.c | 42 ++
mm/damon/ops-common.h | 2 +
mm/damon/paddr.c | 60 +-
mm/damon/sysfs.c | 2 +
12 files changed, 1006 insertions(+), 20 deletions(-)
create mode 100644 Documentation/admin-guide/mm/damon/lru_sort.rst
create mode 100644 mm/damon/lru_sort.c
--
2.25.1
DAMON debugfs interface assumes users will write 'damos_action' value
directly to the 'schemes' file. This makes adding new 'damos_action' in
the middle of its definition breaks the backward compatibility of DAMON
debugfs interface, as values of some 'damos_action' could be changed.
To support backward compatibility in the cases, this commit adds
mappings between the user inputs and 'damos_action' value and makes
DAMON debugfs code uses those.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/dbgfs.c | 64 +++++++++++++++++++++++++++++++++++++-----------
1 file changed, 50 insertions(+), 14 deletions(-)
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
index 5ae810927309..cb8a7e9926a4 100644
--- a/mm/damon/dbgfs.c
+++ b/mm/damon/dbgfs.c
@@ -97,6 +97,31 @@ static ssize_t dbgfs_attrs_write(struct file *file,
return ret;
}
+/*
+ * Return corresponding dbgfs' scheme action value (int) for the given
+ * damos_action if the given damos_action value is valid and supported by
+ * dbgfs, negative error code otherwise.
+ */
+static int damos_action_to_dbgfs_scheme_action(enum damos_action action)
+{
+ switch (action) {
+ case DAMOS_WILLNEED:
+ return 0;
+ case DAMOS_COLD:
+ return 1;
+ case DAMOS_PAGEOUT:
+ return 2;
+ case DAMOS_HUGEPAGE:
+ return 3;
+ case DAMOS_NOHUGEPAGE:
+ return 4;
+ case DAMOS_STAT:
+ return 5;
+ default:
+ return -EINVAL;
+ }
+}
+
static ssize_t sprint_schemes(struct damon_ctx *c, char *buf, ssize_t len)
{
struct damos *s;
@@ -109,7 +134,7 @@ static ssize_t sprint_schemes(struct damon_ctx *c, char *buf, ssize_t len)
s->min_sz_region, s->max_sz_region,
s->min_nr_accesses, s->max_nr_accesses,
s->min_age_region, s->max_age_region,
- s->action,
+ damos_action_to_dbgfs_scheme_action(s->action),
s->quota.ms, s->quota.sz,
s->quota.reset_interval,
s->quota.weight_sz,
@@ -160,18 +185,27 @@ static void free_schemes_arr(struct damos **schemes, ssize_t nr_schemes)
kfree(schemes);
}
-static bool damos_action_valid(int action)
+/*
+ * Return corresponding damos_action for the given dbgfs input for a scheme
+ * action if the input is valid, negative error code otherwise.
+ */
+static enum damos_action dbgfs_scheme_action_to_damos_action(int dbgfs_action)
{
- switch (action) {
- case DAMOS_WILLNEED:
- case DAMOS_COLD:
- case DAMOS_PAGEOUT:
- case DAMOS_HUGEPAGE:
- case DAMOS_NOHUGEPAGE:
- case DAMOS_STAT:
- return true;
+ switch (dbgfs_action) {
+ case 0:
+ return DAMOS_WILLNEED;
+ case 1:
+ return DAMOS_COLD;
+ case 2:
+ return DAMOS_PAGEOUT;
+ case 3:
+ return DAMOS_HUGEPAGE;
+ case 4:
+ return DAMOS_NOHUGEPAGE;
+ case 5:
+ return DAMOS_STAT;
default:
- return false;
+ return -EINVAL;
}
}
@@ -189,7 +223,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len,
int pos = 0, parsed, ret;
unsigned long min_sz, max_sz;
unsigned int min_nr_a, max_nr_a, min_age, max_age;
- unsigned int action;
+ unsigned int action_input;
+ enum damos_action action;
schemes = kmalloc_array(max_nr_schemes, sizeof(scheme),
GFP_KERNEL);
@@ -204,7 +239,7 @@ static struct damos **str_to_schemes(const char *str, ssize_t len,
ret = sscanf(&str[pos],
"%lu %lu %u %u %u %u %u %lu %lu %lu %u %u %u %u %lu %lu %lu %lu%n",
&min_sz, &max_sz, &min_nr_a, &max_nr_a,
- &min_age, &max_age, &action, "a.ms,
+ &min_age, &max_age, &action_input, "a.ms,
"a.sz, "a.reset_interval,
"a.weight_sz, "a.weight_nr_accesses,
"a.weight_age, &wmarks.metric,
@@ -212,7 +247,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len,
&wmarks.low, &parsed);
if (ret != 18)
break;
- if (!damos_action_valid(action))
+ action = dbgfs_scheme_action_to_damos_action(action_input);
+ if ((int)action < 0)
goto fail;
if (min_sz > max_sz || min_nr_a > max_nr_a || min_age > max_age)
--
2.25.1
DAMON debugfs interface assumes users will write 'damos_action' value
directly to the 'schemes' file. This makes adding new 'damos_action' in
the middle of its definition breaks the backward compatibility of DAMON
debugfs interface, as values of some 'damos_action' could be changed.
To mitigate the situation, this commit adds mappings between the user
inputs and 'damos_action' value and makes DAMON debugfs code uses those.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/dbgfs.c | 64 +++++++++++++++++++++++++++++++++++++-----------
1 file changed, 50 insertions(+), 14 deletions(-)
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
index 5ae810927309..cb8a7e9926a4 100644
--- a/mm/damon/dbgfs.c
+++ b/mm/damon/dbgfs.c
@@ -97,6 +97,31 @@ static ssize_t dbgfs_attrs_write(struct file *file,
return ret;
}
+/*
+ * Return corresponding dbgfs' scheme action value (int) for the given
+ * damos_action if the given damos_action value is valid and supported by
+ * dbgfs, negative error code otherwise.
+ */
+static int damos_action_to_dbgfs_scheme_action(enum damos_action action)
+{
+ switch (action) {
+ case DAMOS_WILLNEED:
+ return 0;
+ case DAMOS_COLD:
+ return 1;
+ case DAMOS_PAGEOUT:
+ return 2;
+ case DAMOS_HUGEPAGE:
+ return 3;
+ case DAMOS_NOHUGEPAGE:
+ return 4;
+ case DAMOS_STAT:
+ return 5;
+ default:
+ return -EINVAL;
+ }
+}
+
static ssize_t sprint_schemes(struct damon_ctx *c, char *buf, ssize_t len)
{
struct damos *s;
@@ -109,7 +134,7 @@ static ssize_t sprint_schemes(struct damon_ctx *c, char *buf, ssize_t len)
s->min_sz_region, s->max_sz_region,
s->min_nr_accesses, s->max_nr_accesses,
s->min_age_region, s->max_age_region,
- s->action,
+ damos_action_to_dbgfs_scheme_action(s->action),
s->quota.ms, s->quota.sz,
s->quota.reset_interval,
s->quota.weight_sz,
@@ -160,18 +185,27 @@ static void free_schemes_arr(struct damos **schemes, ssize_t nr_schemes)
kfree(schemes);
}
-static bool damos_action_valid(int action)
+/*
+ * Return corresponding damos_action for the given dbgfs input for a scheme
+ * action if the input is valid, negative error code otherwise.
+ */
+static enum damos_action dbgfs_scheme_action_to_damos_action(int dbgfs_action)
{
- switch (action) {
- case DAMOS_WILLNEED:
- case DAMOS_COLD:
- case DAMOS_PAGEOUT:
- case DAMOS_HUGEPAGE:
- case DAMOS_NOHUGEPAGE:
- case DAMOS_STAT:
- return true;
+ switch (dbgfs_action) {
+ case 0:
+ return DAMOS_WILLNEED;
+ case 1:
+ return DAMOS_COLD;
+ case 2:
+ return DAMOS_PAGEOUT;
+ case 3:
+ return DAMOS_HUGEPAGE;
+ case 4:
+ return DAMOS_NOHUGEPAGE;
+ case 5:
+ return DAMOS_STAT;
default:
- return false;
+ return -EINVAL;
}
}
@@ -189,7 +223,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len,
int pos = 0, parsed, ret;
unsigned long min_sz, max_sz;
unsigned int min_nr_a, max_nr_a, min_age, max_age;
- unsigned int action;
+ unsigned int action_input;
+ enum damos_action action;
schemes = kmalloc_array(max_nr_schemes, sizeof(scheme),
GFP_KERNEL);
@@ -204,7 +239,7 @@ static struct damos **str_to_schemes(const char *str, ssize_t len,
ret = sscanf(&str[pos],
"%lu %lu %u %u %u %u %u %lu %lu %lu %u %u %u %u %lu %lu %lu %lu%n",
&min_sz, &max_sz, &min_nr_a, &max_nr_a,
- &min_age, &max_age, &action, "a.ms,
+ &min_age, &max_age, &action_input, "a.ms,
"a.sz, "a.reset_interval,
"a.weight_sz, "a.weight_nr_accesses,
"a.weight_age, &wmarks.metric,
@@ -212,7 +247,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len,
&wmarks.low, &parsed);
if (ret != 18)
break;
- if (!damos_action_valid(action))
+ action = dbgfs_scheme_action_to_damos_action(action_input);
+ if ((int)action < 0)
goto fail;
if (min_sz > max_sz || min_nr_a > max_nr_a || min_age > max_age)
--
2.25.1
This commit documents the 'lru_prio' scheme action for DAMON sysfs
interface.
Signed-off-by: SeongJae Park <[email protected]>
---
Documentation/admin-guide/mm/damon/usage.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst
index 1bb7b72414b2..af4e15ee81cd 100644
--- a/Documentation/admin-guide/mm/damon/usage.rst
+++ b/Documentation/admin-guide/mm/damon/usage.rst
@@ -264,6 +264,7 @@ that can be written to and read from the file and their meaning are as below.
- ``pageout``: Call ``madvise()`` for the region with ``MADV_PAGEOUT``
- ``hugepage``: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``
- ``nohugepage``: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``
+ - ``lru_prio``: Prioritize the region on its LRU lists.
- ``stat``: Do nothing but count the statistics
schemes/<N>/access_pattern/
--
2.25.1
This commit documents the usage of DAMON_LRU_SORT for admins.
Signed-off-by: SeongJae Park <[email protected]>
---
Documentation/admin-guide/mm/damon/index.rst | 1 +
.../admin-guide/mm/damon/lru_sort.rst | 294 ++++++++++++++++++
2 files changed, 295 insertions(+)
create mode 100644 Documentation/admin-guide/mm/damon/lru_sort.rst
diff --git a/Documentation/admin-guide/mm/damon/index.rst b/Documentation/admin-guide/mm/damon/index.rst
index 61aff88347f3..53762770e0e4 100644
--- a/Documentation/admin-guide/mm/damon/index.rst
+++ b/Documentation/admin-guide/mm/damon/index.rst
@@ -14,3 +14,4 @@ optimize those.
start
usage
reclaim
+ lru_sort
diff --git a/Documentation/admin-guide/mm/damon/lru_sort.rst b/Documentation/admin-guide/mm/damon/lru_sort.rst
new file mode 100644
index 000000000000..c09cace80651
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/lru_sort.rst
@@ -0,0 +1,294 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=============================
+DAMON-based LRU-lists Sorting
+=============================
+
+DAMON-based LRU-lists Sorting (DAMON_LRU_SORT) is a static kernel module that
+aimed to be used for proactive and lightweight data access pattern based
+(de)prioritization of pages on their LRU-lists for making LRU-lists a more
+trusworthy data access pattern source.
+
+Where Proactive LRU-lists Sorting is Required?
+==============================================
+
+As page-granularity access checking overhead could be significant on huge
+systems, LRU lists are normally not proactively sorted but partially and
+reactively sorted for special events including specific user requests, system
+calls and memory pressure. As a result, LRU lists are sometimes not so
+perfectly prepared to be used as a trustworthy access pattern source for some
+situations including reclamation target pages selection under sudden memory
+pressure.
+
+Because DAMON can identify access patterns of best-effort accuracy while
+inducing only user-specified range of overhead, proactively running
+DAMON_LRU_SORT could be helpful for making LRU lists more trustworthy access
+pattern source with low and controlled overhead.
+
+How It Works?
+=============
+
+DAMON_LRU_SORT finds hot pages (pages of memory regions that showing access
+rates that higher than a user-specified threshold) and cold pages (pages of
+memory regions that showing no access for a time that longer than a
+user-specified threshold) using DAMON, and prioritizes hot pages while
+deprioritizing cold pages on their LRU-lists. To avoid it consuming too much
+CPU for the prioritizations, a CPU time usage limit can be configured. Under
+the limit, it prioritizes and deprioritizes more hot and cold pages first,
+respectively. System administrators can also configure under what situation
+this scheme should automatically activated and deactivated with three memory
+pressure watermarks.
+
+Its default parameters for hotness/coldness thresholds and CPU quota limit are
+conservatively chosen. That is, the module under its default parameters could
+be widely used without harm for common situations while providing a level of
+benefits for systems having clear hot/cold access patterns under memory
+pressure while consuming only a limited small portion of CPU time.
+
+Interface: Module Parameters
+============================
+
+To use this feature, you should first ensure your system is running on a kernel
+that is built with ``CONFIG_DAMON_LRU_SORT=y``.
+
+To let sysadmins enable or disable it and tune for the given system,
+DAMON_LRU_SORT utilizes module parameters. That is, you can put
+``damon_lru_sort.<parameter>=<value>`` on the kernel boot command line or write
+proper values to ``/sys/modules/damon_lru_sort/parameters/<parameter>`` files.
+
+Below are the description of each parameter.
+
+enabled
+-------
+
+Enable or disable DAMON_LRU_SORT.
+
+You can enable DAMON_LRU_SORT by setting the value of this parameter as ``Y``.
+Setting it as ``N`` disables DAMON_LRU_SORT. Note that DAMON_LRU_SORT could do
+no real monitoring and LRU-lists sorting due to the watermarks-based activation
+condition. Refer to below descriptions for the watermarks parameter for this.
+
+commit_inputs
+-------------
+
+Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``.
+
+Input parameters that updated while DAMON_LRU_SORT is running are not applied
+by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT reads values
+of parametrs except ``enabled`` again. Once the re-reading is done, this
+parameter is set as ``N``. If invalid parameters are found while the
+re-reading, DAMON_LRU_SORT will be disabled.
+
+hot_thres_access_freq
+---------------------
+
+Access frequency threshold for hot memory regions identification in permil.
+
+If a memory region is accessed in frequency of this or higher, DAMON_LRU_SORT
+identifies the region as hot, and mark it as accessed on the LRU list, so that
+it could not be reclaimed under memory pressure. 50% by default.
+
+cold_min_age
+------------
+
+Time threshold for cold memory regions identification in microseconds.
+
+If a memory region is not accessed for this or longer time, DAMON_LRU_SORT
+identifies the region as cold, and mark it as unaccessed on the LRU list, so
+that it could be reclaimed first under memory pressure. 120 seconds by
+default.
+
+quota_ms
+--------
+
+Limit of time for trying the LRU lists sorting in milliseconds.
+
+DAMON_LRU_SORT tries to use only up to this time within a time window
+(quota_reset_interval_ms) for trying LRU lists sorting. This can be used
+for limiting CPU consumption of DAMON_LRU_SORT. If the value is zero, the
+limit is disabled.
+
+10 ms by default.
+
+quota_reset_interval_ms
+-----------------------
+
+The time quota charge reset interval in milliseconds.
+
+The charge reset interval for the quota of time (quota_ms). That is,
+DAMON_LRU_SORT does not try LRU-lists sorting for more than quota_ms
+milliseconds or quota_sz bytes within quota_reset_interval_ms milliseconds.
+
+1 second by default.
+
+wmarks_interval
+---------------
+
+The watermarks check time interval in microseconds.
+
+Minimal time to wait before checking the watermarks, when DAMON_LRU_SORT is
+enabled but inactive due to its watermarks rule. 5 seconds by default.
+
+wmarks_high
+-----------
+
+Free memory rate (per thousand) for the high watermark.
+
+If free memory of the system in bytes per thousand bytes is higher than this,
+DAMON_LRU_SORT becomes inactive, so it does nothing but periodically checks the
+watermarks. 200 (20%) by default.
+
+wmarks_mid
+----------
+
+Free memory rate (per thousand) for the middle watermark.
+
+If free memory of the system in bytes per thousand bytes is between this and
+the low watermark, DAMON_LRU_SORT becomes active, so starts the monitoring and
+the LRU-lists sorting. 150 (15%) by default.
+
+wmarks_low
+----------
+
+Free memory rate (per thousand) for the low watermark.
+
+If free memory of the system in bytes per thousand bytes is lower than this,
+DAMON_LRU_SORT becomes inactive, so it does nothing but periodically checks the
+watermarks. 50 (5%) by default.
+
+sample_interval
+---------------
+
+Sampling interval for the monitoring in microseconds.
+
+The sampling interval of DAMON for the cold memory monitoring. Please refer to
+the DAMON documentation (:doc:`usage`) for more detail. 5ms by default.
+
+aggr_interval
+-------------
+
+Aggregation interval for the monitoring in microseconds.
+
+The aggregation interval of DAMON for the cold memory monitoring. Please
+refer to the DAMON documentation (:doc:`usage`) for more detail. 100ms by
+default.
+
+min_nr_regions
+--------------
+
+Minimum number of monitoring regions.
+
+The minimal number of monitoring regions of DAMON for the cold memory
+monitoring. This can be used to set lower-bound of the monitoring quality.
+But, setting this too high could result in increased monitoring overhead.
+Please refer to the DAMON documentation (:doc:`usage`) for more detail. 10 by
+default.
+
+max_nr_regions
+--------------
+
+Maximum number of monitoring regions.
+
+The maximum number of monitoring regions of DAMON for the cold memory
+monitoring. This can be used to set upper-bound of the monitoring overhead.
+However, setting this too low could result in bad monitoring quality. Please
+refer to the DAMON documentation (:doc:`usage`) for more detail. 1000 by
+defaults.
+
+monitor_region_start
+--------------------
+
+Start of target memory region in physical address.
+
+The start physical address of memory region that DAMON_LRU_SORT will do work
+against. By default, biggest System RAM is used as the region.
+
+monitor_region_end
+------------------
+
+End of target memory region in physical address.
+
+The end physical address of memory region that DAMON_LRU_SORT will do work
+against. By default, biggest System RAM is used as the region.
+
+kdamond_pid
+-----------
+
+PID of the DAMON thread.
+
+If DAMON_LRU_SORT is enabled, this becomes the PID of the worker thread. Else,
+-1.
+
+nr_lru_sort_tried_hot_regions
+-----------------------------
+
+Number of hot memory regions that tried to be LRU-sorted.
+
+bytes_lru_sort_tried_hot_regions
+--------------------------------
+
+Total bytes of hot memory regions that tried to be LRU-sorted.
+
+nr_lru_sorted_hot_regions
+-------------------------
+
+Number of hot memory regions that successfully be LRU-sorted.
+
+bytes_lru_sorted_hot_regions
+----------------------------
+
+Total bytes of hot memory regions that successfully be LRU-sorted.
+
+nr_hot_quota_exceeds
+--------------------
+
+Number of times that the time quota limit for hot regions have exceeded.
+
+nr_lru_sort_tried_cold_regions
+------------------------------
+
+Number of cold memory regions that tried to be LRU-sorted.
+
+bytes_lru_sort_tried_cold_regions
+---------------------------------
+
+Total bytes of cold memory regions that tried to be LRU-sorted.
+
+nr_lru_sorted_cold_regions
+--------------------------
+
+Number of cold memory regions that successfully be LRU-sorted.
+
+bytes_lru_sorted_cold_regions
+-----------------------------
+
+Total bytes of cold memory regions that successfully be LRU-sorted.
+
+nr_cold_quota_exceeds
+---------------------
+
+Number of times that the time quota limit for cold regions have exceeded.
+
+Example
+=======
+
+Below runtime example commands make DAMON_LRU_SORT to find memory regions
+having >=50% access frequency and LRU-prioritize while LRU-deprioritizing
+memory regions that not accessed for 120 seconds. The prioritization and
+deprioritization is limited to be done using only up to 1% CPU time to avoid
+DAMON_LRU_SORT consuming too much CPU time for the (de)prioritization. It also
+asks DAMON_LRU_SORT to do nothing if the system's free memory rate is more than
+50%, but start the real works if it becomes lower than 40%. If DAMON_RECLAIM
+doesn't make progress and therefore the free memory rate becomes lower than
+20%, it asks DAMON_LRU_SORT to do nothing again, so that we can fall back to
+the LRU-list based page granularity reclamation. ::
+
+ # cd /sys/modules/damon_lru_sort/parameters
+ # echo 500 > hot_thres_access_freq
+ # echo 120000000 > cold_min_age
+ # echo 10 > quota_ms
+ # echo 1000 > quota_reset_interval_ms
+ # echo 500 > wmarks_high
+ # echo 400 > wmarks_mid
+ # echo 200 > wmarks_low
+ # echo Y > enabled
--
2.25.1
This commit documents the 'LRU_DEPRIO' scheme action for DAMON sysfs
interface.`
Signed-off-by: SeongJae Park <[email protected]>
---
Documentation/admin-guide/mm/damon/usage.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst
index af4e15ee81cd..d822bf6355ce 100644
--- a/Documentation/admin-guide/mm/damon/usage.rst
+++ b/Documentation/admin-guide/mm/damon/usage.rst
@@ -265,6 +265,7 @@ that can be written to and read from the file and their meaning are as below.
- ``hugepage``: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``
- ``nohugepage``: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``
- ``lru_prio``: Prioritize the region on its LRU lists.
+ - ``lru_deprio``: Deprioritize the region on its LRU lists.
- ``stat``: Do nothing but count the statistics
schemes/<N>/access_pattern/
--
2.25.1
Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
the hotness/coldness thresholds, CPU quota, and watermarks could be
challenging for some users. To make the scheme easy to be used without
complex tuning for common situations, this commit implements a static
kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
'LRU_DEPRIO' DAMOS actions.
It proactively sorts LRU-lists using DAMON with conservatively chosen
default values of the parameters. That is, the module under its default
parameters will make no harm for common situations but provide some
level of efficiency improvements for systems having clear hot/cold
access pattern under a level of memory pressure while consuming only a
limited small portion of CPU time.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/Kconfig | 8 +
mm/damon/Makefile | 1 +
mm/damon/lru_sort.c | 546 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 555 insertions(+)
create mode 100644 mm/damon/lru_sort.c
diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
index 9b559c76d6dd..66265e3a9c65 100644
--- a/mm/damon/Kconfig
+++ b/mm/damon/Kconfig
@@ -92,4 +92,12 @@ config DAMON_RECLAIM
reclamation under light memory pressure, while the traditional page
scanning-based reclamation is used for heavy pressure.
+config DAMON_LRU_SORT
+ bool "Build DAMON-based LRU-lists sorting (DAMON_LRU_SORT)"
+ depends on DAMON_PADDR
+ help
+ This builds the DAMON-based LRU-lists sorting subsystem. It tries to
+ protect frequently accessed (hot) pages while rarely accessed (cold)
+ pages reclaimed first under memory pressure.
+
endmenu
diff --git a/mm/damon/Makefile b/mm/damon/Makefile
index dbf7190b4144..3e6b8ad73858 100644
--- a/mm/damon/Makefile
+++ b/mm/damon/Makefile
@@ -6,3 +6,4 @@ obj-$(CONFIG_DAMON_PADDR) += ops-common.o paddr.o
obj-$(CONFIG_DAMON_SYSFS) += sysfs.o
obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
obj-$(CONFIG_DAMON_RECLAIM) += reclaim.o
+obj-$(CONFIG_DAMON_LRU_SORT) += lru_sort.o
diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c
new file mode 100644
index 000000000000..c276736a071c
--- /dev/null
+++ b/mm/damon/lru_sort.c
@@ -0,0 +1,546 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DAMON-based LRU-lists Sorting
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#define pr_fmt(fmt) "damon-lru-sort: " fmt
+
+#include <linux/damon.h>
+#include <linux/ioport.h>
+#include <linux/module.h>
+#include <linux/sched.h>
+#include <linux/workqueue.h>
+
+#ifdef MODULE_PARAM_PREFIX
+#undef MODULE_PARAM_PREFIX
+#endif
+#define MODULE_PARAM_PREFIX "damon_lru_sort."
+
+/*
+ * Enable or disable DAMON_LRU_SORT.
+ *
+ * You can enable DAMON_LRU_SORT by setting the value of this parameter as
+ * ``Y``. Setting it as ``N`` disables DAMON_LRU_SORT. Note that
+ * DAMON_LRU_SORT could do no real monitoring and LRU-lists sorting due to the
+ * watermarks-based activation condition. Refer to below descriptions for the
+ * watermarks parameter for this.
+ */
+static bool enabled __read_mostly;
+
+/*
+ * Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``.
+ *
+ * Input parameters that updated while DAMON_LRU_SORT is running are not
+ * applied by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT
+ * reads values of parametrs except ``enabled`` again. Once the re-reading is
+ * done, this parameter is set as ``N``. If invalid parameters are found while
+ * the re-reading, DAMON_LRU_SORT will be disabled.
+ */
+static bool commit_inputs __read_mostly;
+module_param(commit_inputs, bool, 0600);
+
+/*
+ * Access frequency threshold for hot memory regions identification in permil.
+ *
+ * If a memory region is accessed in frequency of this or higher,
+ * DAMON_LRU_SORT identifies the region as hot, and mark it as accessed on the
+ * LRU list, so that it could not be reclaimed under memory pressure. 50% by
+ * default.
+ */
+static unsigned long hot_thres_access_freq = 500;
+module_param(hot_thres_access_freq, ulong, 0600);
+
+/*
+ * Time threshold for cold memory regions identification in microseconds.
+ *
+ * If a memory region is not accessed for this or longer time, DAMON_LRU_SORT
+ * identifies the region as cold, and mark it as unaccessed on the LRU list, so
+ * that it could be reclaimed first under memory pressure. 120 seconds by
+ * default.
+ */
+static unsigned long cold_min_age __read_mostly = 120000000;
+module_param(cold_min_age, ulong, 0600);
+
+/*
+ * Limit of time for trying the LRU lists sorting in milliseconds.
+ *
+ * DAMON_LRU_SORT tries to use only up to this time within a time window
+ * (quota_reset_interval_ms) for trying LRU lists sorting. This can be used
+ * for limiting CPU consumption of DAMON_LRU_SORT. If the value is zero, the
+ * limit is disabled.
+ *
+ * 10 ms by default.
+ */
+static unsigned long quota_ms __read_mostly = 10;
+module_param(quota_ms, ulong, 0600);
+
+/*
+ * The time quota charge reset interval in milliseconds.
+ *
+ * The charge reset interval for the quota of time (quota_ms). That is,
+ * DAMON_LRU_SORT does not try LRU-lists sorting for more than quota_ms
+ * milliseconds or quota_sz bytes within quota_reset_interval_ms milliseconds.
+ *
+ * 1 second by default.
+ */
+static unsigned long quota_reset_interval_ms __read_mostly = 1000;
+module_param(quota_reset_interval_ms, ulong, 0600);
+
+/*
+ * The watermarks check time interval in microseconds.
+ *
+ * Minimal time to wait before checking the watermarks, when DAMON_LRU_SORT is
+ * enabled but inactive due to its watermarks rule. 5 seconds by default.
+ */
+static unsigned long wmarks_interval __read_mostly = 5000000;
+module_param(wmarks_interval, ulong, 0600);
+
+/*
+ * Free memory rate (per thousand) for the high watermark.
+ *
+ * If free memory of the system in bytes per thousand bytes is higher than
+ * this, DAMON_LRU_SORT becomes inactive, so it does nothing but periodically
+ * checks the watermarks. 200 (20%) by default.
+ */
+static unsigned long wmarks_high __read_mostly = 200;
+module_param(wmarks_high, ulong, 0600);
+
+/*
+ * Free memory rate (per thousand) for the middle watermark.
+ *
+ * If free memory of the system in bytes per thousand bytes is between this and
+ * the low watermark, DAMON_LRU_SORT becomes active, so starts the monitoring
+ * and the LRU-lists sorting. 150 (15%) by default.
+ */
+static unsigned long wmarks_mid __read_mostly = 150;
+module_param(wmarks_mid, ulong, 0600);
+
+/*
+ * Free memory rate (per thousand) for the low watermark.
+ *
+ * If free memory of the system in bytes per thousand bytes is lower than this,
+ * DAMON_LRU_SORT becomes inactive, so it does nothing but periodically checks
+ * the watermarks. 50 (5%) by default.
+ */
+static unsigned long wmarks_low __read_mostly = 50;
+module_param(wmarks_low, ulong, 0600);
+
+/*
+ * Sampling interval for the monitoring in microseconds.
+ *
+ * The sampling interval of DAMON for the hot/cold memory monitoring. Please
+ * refer to the DAMON documentation for more detail. 5 ms by default.
+ */
+static unsigned long sample_interval __read_mostly = 5000;
+module_param(sample_interval, ulong, 0600);
+
+/*
+ * Aggregation interval for the monitoring in microseconds.
+ *
+ * The aggregation interval of DAMON for the hot/cold memory monitoring.
+ * Please refer to the DAMON documentation for more detail. 100 ms by default.
+ */
+static unsigned long aggr_interval __read_mostly = 100000;
+module_param(aggr_interval, ulong, 0600);
+
+/*
+ * Minimum number of monitoring regions.
+ *
+ * The minimal number of monitoring regions of DAMON for the hot/cold memory
+ * monitoring. This can be used to set lower-bound of the monitoring quality.
+ * But, setting this too high could result in increased monitoring overhead.
+ * Please refer to the DAMON documentation for more detail. 10 by default.
+ */
+static unsigned long min_nr_regions __read_mostly = 10;
+module_param(min_nr_regions, ulong, 0600);
+
+/*
+ * Maximum number of monitoring regions.
+ *
+ * The maximum number of monitoring regions of DAMON for the hot/cold memory
+ * monitoring. This can be used to set upper-bound of the monitoring overhead.
+ * However, setting this too low could result in bad monitoring quality.
+ * Please refer to the DAMON documentation for more detail. 1000 by default.
+ */
+static unsigned long max_nr_regions __read_mostly = 1000;
+module_param(max_nr_regions, ulong, 0600);
+
+/*
+ * Start of the target memory region in physical address.
+ *
+ * The start physical address of memory region that DAMON_LRU_SORT will do work
+ * against. By default, biggest System RAM is used as the region.
+ */
+static unsigned long monitor_region_start __read_mostly;
+module_param(monitor_region_start, ulong, 0600);
+
+/*
+ * End of the target memory region in physical address.
+ *
+ * The end physical address of memory region that DAMON_LRU_SORT will do work
+ * against. By default, biggest System RAM is used as the region.
+ */
+static unsigned long monitor_region_end __read_mostly;
+module_param(monitor_region_end, ulong, 0600);
+
+/*
+ * PID of the DAMON thread
+ *
+ * If DAMON_LRU_SORT is enabled, this becomes the PID of the worker thread.
+ * Else, -1.
+ */
+static int kdamond_pid __read_mostly = -1;
+module_param(kdamond_pid, int, 0400);
+
+/*
+ * Number of hot memory regions that tried to be LRU-sorted.
+ */
+static unsigned long nr_lru_sort_tried_hot_regions __read_mostly;
+module_param(nr_lru_sort_tried_hot_regions, ulong, 0400);
+
+/*
+ * Total bytes of hot memory regions that tried to be LRU-sorted.
+ */
+static unsigned long bytes_lru_sort_tried_hot_regions __read_mostly;
+module_param(bytes_lru_sort_tried_hot_regions, ulong, 0400);
+
+/*
+ * Number of hot memory regions that successfully be LRU-sorted.
+ */
+static unsigned long nr_lru_sorted_hot_regions __read_mostly;
+module_param(nr_lru_sorted_hot_regions, ulong, 0400);
+
+/*
+ * Total bytes of hot memory regions that successfully be LRU-sorted.
+ */
+static unsigned long bytes_lru_sorted_hot_regions __read_mostly;
+module_param(bytes_lru_sorted_hot_regions, ulong, 0400);
+
+/*
+ * Number of times that the time quota limit for hot regions have exceeded
+ */
+static unsigned long nr_hot_quota_exceeds __read_mostly;
+module_param(nr_hot_quota_exceeds, ulong, 0400);
+
+/*
+ * Number of cold memory regions that tried to be LRU-sorted.
+ */
+static unsigned long nr_lru_sort_tried_cold_regions __read_mostly;
+module_param(nr_lru_sort_tried_cold_regions, ulong, 0400);
+
+/*
+ * Total bytes of cold memory regions that tried to be LRU-sorted.
+ */
+static unsigned long bytes_lru_sort_tried_cold_regions __read_mostly;
+module_param(bytes_lru_sort_tried_cold_regions, ulong, 0400);
+
+/*
+ * Number of cold memory regions that successfully be LRU-sorted.
+ */
+static unsigned long nr_lru_sorted_cold_regions __read_mostly;
+module_param(nr_lru_sorted_cold_regions, ulong, 0400);
+
+/*
+ * Total bytes of cold memory regions that successfully be LRU-sorted.
+ */
+static unsigned long bytes_lru_sorted_cold_regions __read_mostly;
+module_param(bytes_lru_sorted_cold_regions, ulong, 0400);
+
+/*
+ * Number of times that the time quota limit for cold regions have exceeded
+ */
+static unsigned long nr_cold_quota_exceeds __read_mostly;
+module_param(nr_cold_quota_exceeds, ulong, 0400);
+
+static struct damon_ctx *ctx;
+static struct damon_target *target;
+
+struct damon_lru_sort_ram_walk_arg {
+ unsigned long start;
+ unsigned long end;
+};
+
+static int walk_system_ram(struct resource *res, void *arg)
+{
+ struct damon_lru_sort_ram_walk_arg *a = arg;
+
+ if (a->end - a->start < resource_size(res)) {
+ a->start = res->start;
+ a->end = res->end;
+ }
+ return 0;
+}
+
+/*
+ * Find biggest 'System RAM' resource and store its start and end address in
+ * @start and @end, respectively. If no System RAM is found, returns false.
+ */
+static bool get_monitoring_region(unsigned long *start, unsigned long *end)
+{
+ struct damon_lru_sort_ram_walk_arg arg = {};
+
+ walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram);
+ if (arg.end <= arg.start)
+ return false;
+
+ *start = arg.start;
+ *end = arg.end;
+ return true;
+}
+
+/* Create a DAMON-based operation scheme for hot memory regions */
+static struct damos *damon_lru_sort_new_hot_scheme(unsigned int hot_thres)
+{
+ struct damos_watermarks wmarks = {
+ .metric = DAMOS_WMARK_FREE_MEM_RATE,
+ .interval = wmarks_interval,
+ .high = wmarks_high,
+ .mid = wmarks_mid,
+ .low = wmarks_low,
+ };
+ struct damos_quota quota = {
+ /*
+ * Do not try LRU-lists sorting of hot pages for more than half
+ * of quota_ms milliseconds within quota_reset_interval_ms.
+ */
+ .ms = quota_ms / 2,
+ .sz = 0,
+ .reset_interval = quota_reset_interval_ms,
+ /* Within the quota, mark hotter regions accessed first. */
+ .weight_sz = 0,
+ .weight_nr_accesses = 1,
+ .weight_age = 0,
+ };
+ struct damos *scheme = damon_new_scheme(
+ /* Find regions having PAGE_SIZE or larger size */
+ PAGE_SIZE, ULONG_MAX,
+ /* and accessed for more than the threshold */
+ hot_thres, UINT_MAX,
+ /* no matter its age */
+ 0, UINT_MAX,
+ /* prioritize those on LRU lists, as soon as found */
+ DAMOS_LRU_PRIO,
+ /* under the quota. */
+ "a,
+ /* (De)activate this according to the watermarks. */
+ &wmarks);
+
+ return scheme;
+}
+
+/* Create a DAMON-based operation scheme for cold memory regions */
+static struct damos *damon_lru_sort_new_cold_scheme(unsigned int cold_thres)
+{
+ struct damos_watermarks wmarks = {
+ .metric = DAMOS_WMARK_FREE_MEM_RATE,
+ .interval = wmarks_interval,
+ .high = wmarks_high,
+ .mid = wmarks_mid,
+ .low = wmarks_low,
+ };
+ struct damos_quota quota = {
+ /*
+ * Do not try LRU-lists sorting of cold pages for more than
+ * half of quota_ms milliseconds within
+ * quota_reset_interval_ms.
+ */
+ .ms = quota_ms / 2,
+ .sz = 0,
+ .reset_interval = quota_reset_interval_ms,
+ /* Within the quota, mark colder regions not accessed first. */
+ .weight_sz = 0,
+ .weight_nr_accesses = 0,
+ .weight_age = 1,
+ };
+ struct damos *scheme = damon_new_scheme(
+ /* Find regions having PAGE_SIZE or larger size */
+ PAGE_SIZE, ULONG_MAX,
+ /* and not accessed at all */
+ 0, 0,
+ /* for cold_thres or more micro-seconds, and */
+ cold_thres, UINT_MAX,
+ /* mark those as not accessed, as soon as found */
+ DAMOS_LRU_DEPRIO,
+ /* under the quota. */
+ "a,
+ /* (De)activate this according to the watermarks. */
+ &wmarks);
+
+ return scheme;
+}
+
+static int damon_lru_sort_apply_parameters(void)
+{
+ struct damos *scheme, *next_scheme;
+ struct damon_addr_range addr_range;
+ unsigned int hot_thres, cold_thres;
+ int err = 0;
+
+ err = damon_set_attrs(ctx, sample_interval, aggr_interval, 0,
+ min_nr_regions, max_nr_regions);
+ if (err)
+ return err;
+
+ /* free previously set schemes */
+ damon_for_each_scheme_safe(scheme, next_scheme, ctx)
+ damon_destroy_scheme(scheme);
+
+ /* aggr_interval / sample_interval is the maximum nr_accesses */
+ hot_thres = aggr_interval / sample_interval * hot_thres_access_freq /
+ 1000;
+ scheme = damon_lru_sort_new_hot_scheme(hot_thres);
+ if (!scheme)
+ return -ENOMEM;
+ damon_add_scheme(ctx, scheme);
+
+ cold_thres = cold_min_age / aggr_interval;
+ scheme = damon_lru_sort_new_cold_scheme(cold_thres);
+ if (!scheme)
+ return -ENOMEM;
+ damon_add_scheme(ctx, scheme);
+
+ if (monitor_region_start > monitor_region_end)
+ return -EINVAL;
+ if (!monitor_region_start && !monitor_region_end &&
+ !get_monitoring_region(&monitor_region_start,
+ &monitor_region_end))
+ return -EINVAL;
+ addr_range.start = monitor_region_start;
+ addr_range.end = monitor_region_end;
+ return damon_set_regions(target, &addr_range, 1);
+}
+
+static int damon_lru_sort_turn(bool on)
+{
+ int err;
+
+ if (!on) {
+ err = damon_stop(&ctx, 1);
+ if (!err)
+ kdamond_pid = -1;
+ return err;
+ }
+
+ err = damon_lru_sort_apply_parameters();
+ if (err)
+ return err;
+
+ err = damon_start(&ctx, 1, true);
+ if (err)
+ return err;
+ kdamond_pid = ctx->kdamond->pid;
+ return 0;
+}
+
+static struct delayed_work damon_lru_sort_timer;
+static void damon_lru_sort_timer_fn(struct work_struct *work)
+{
+ static bool last_enabled;
+ bool now_enabled;
+
+ now_enabled = enabled;
+ if (last_enabled != now_enabled) {
+ if (!damon_lru_sort_turn(now_enabled))
+ last_enabled = now_enabled;
+ else
+ enabled = last_enabled;
+ }
+}
+static DECLARE_DELAYED_WORK(damon_lru_sort_timer, damon_lru_sort_timer_fn);
+
+static bool damon_lru_sort_initialized;
+
+static int damon_lru_sort_enabled_store(const char *val,
+ const struct kernel_param *kp)
+{
+ int rc = param_set_bool(val, kp);
+
+ if (rc < 0)
+ return rc;
+
+ if (!damon_lru_sort_initialized)
+ return rc;
+
+ schedule_delayed_work(&damon_lru_sort_timer, 0);
+
+ return 0;
+}
+
+static const struct kernel_param_ops enabled_param_ops = {
+ .set = damon_lru_sort_enabled_store,
+ .get = param_get_bool,
+};
+
+module_param_cb(enabled, &enabled_param_ops, &enabled, 0600);
+MODULE_PARM_DESC(enabled,
+ "Enable or disable DAMON_LRU_SORT (default: disabled)");
+
+static int damon_lru_sort_handle_commit_inputs(void)
+{
+ int err;
+
+ if (!commit_inputs)
+ return 0;
+
+ err = damon_lru_sort_apply_parameters();
+ commit_inputs = false;
+ return err;
+}
+
+static int damon_lru_sort_after_aggregation(struct damon_ctx *c)
+{
+ struct damos *s;
+
+ /* update the stats parameter */
+ damon_for_each_scheme(s, c) {
+ if (s->action == DAMOS_LRU_PRIO) {
+ nr_lru_sort_tried_hot_regions = s->stat.nr_tried;
+ bytes_lru_sort_tried_hot_regions = s->stat.sz_tried;
+ nr_lru_sorted_hot_regions = s->stat.nr_applied;
+ bytes_lru_sorted_hot_regions = s->stat.sz_applied;
+ nr_hot_quota_exceeds = s->stat.qt_exceeds;
+ } else if (s->action == DAMOS_LRU_DEPRIO) {
+ nr_lru_sort_tried_cold_regions = s->stat.nr_tried;
+ bytes_lru_sort_tried_cold_regions = s->stat.sz_tried;
+ nr_lru_sorted_cold_regions = s->stat.nr_applied;
+ bytes_lru_sorted_cold_regions = s->stat.sz_applied;
+ nr_cold_quota_exceeds = s->stat.qt_exceeds;
+ }
+ }
+
+ return damon_lru_sort_handle_commit_inputs();
+}
+
+static int damon_lru_sort_after_wmarks_check(struct damon_ctx *c)
+{
+ return damon_lru_sort_handle_commit_inputs();
+}
+
+static int __init damon_lru_sort_init(void)
+{
+ ctx = damon_new_ctx();
+ if (!ctx)
+ return -ENOMEM;
+
+ if (damon_select_ops(ctx, DAMON_OPS_PADDR))
+ return -EINVAL;
+
+ ctx->callback.after_wmarks_check = damon_lru_sort_after_wmarks_check;
+ ctx->callback.after_aggregation = damon_lru_sort_after_aggregation;
+
+ target = damon_new_target();
+ if (!target) {
+ damon_destroy_ctx(ctx);
+ return -ENOMEM;
+ }
+ damon_add_target(ctx, target);
+
+ schedule_delayed_work(&damon_lru_sort_timer, 0);
+
+ damon_lru_sort_initialized = true;
+ return 0;
+}
+
+module_init(damon_lru_sort_init);
--
2.25.1
This commit adds a new DAMON-based operation scheme action called
'LRU_DEPRIO' for physical address space. The action deprioritizes pages
in the memory area of the target access pattern on their LRU lists.
This is hence supposed to be used for rarely accessed (cold) memory
regions so that cold pages could be more likely reclaimed first under
memory pressure. Internally, it simply calls 'lru_deactivate()'.
Using this with 'LRU_PRIO' action for hot pages, users can proactively
sort LRU lists based on the access pattern. That is, it can make the
LRU lists somewhat more trustworthy source of access temperature. As a
result, efficiency of LRU-lists based mechanisms including the
reclamation target selection could be improved.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/damon.h | 2 ++
mm/damon/paddr.c | 20 ++++++++++++++++++++
mm/damon/sysfs.c | 1 +
3 files changed, 23 insertions(+)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 4c64e03e94d8..7b1f4a488230 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -87,6 +87,7 @@ struct damon_target {
* @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
* @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
* @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.
+ * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.
* @DAMOS_STAT: Do nothing but count the stat.
* @NR_DAMOS_ACTIONS: Total number of DAMOS actions
*/
@@ -97,6 +98,7 @@ enum damos_action {
DAMOS_HUGEPAGE,
DAMOS_NOHUGEPAGE,
DAMOS_LRU_PRIO,
+ DAMOS_LRU_DEPRIO,
DAMOS_STAT, /* Do nothing but only record the stat */
NR_DAMOS_ACTIONS,
};
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index f145b1d51e13..dc131c6a5403 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -249,6 +249,22 @@ static unsigned long damon_pa_mark_accessed(struct damon_region *r)
return applied * PAGE_SIZE;
}
+static unsigned long damon_pa_deactivate_pages(struct damon_region *r)
+{
+ unsigned long addr, applied = 0;
+
+ for (addr = r->ar.start; addr < r->ar.end; addr += PAGE_SIZE) {
+ struct page *page = damon_get_page(PHYS_PFN(addr));
+
+ if (!page)
+ continue;
+ deactivate_page(page);
+ put_page(page);
+ applied++;
+ }
+ return applied * PAGE_SIZE;
+}
+
static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx,
struct damon_target *t, struct damon_region *r,
struct damos *scheme)
@@ -258,6 +274,8 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx,
return damon_pa_pageout(r);
case DAMOS_LRU_PRIO:
return damon_pa_mark_accessed(r);
+ case DAMOS_LRU_DEPRIO:
+ return damon_pa_deactivate_pages(r);
default:
break;
}
@@ -273,6 +291,8 @@ static int damon_pa_scheme_score(struct damon_ctx *context,
return damon_pageout_score(context, r, scheme);
case DAMOS_LRU_PRIO:
return damon_hot_score(context, r, scheme);
+ case DAMOS_LRU_DEPRIO:
+ return damon_pageout_score(context, r, scheme);
default:
break;
}
diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c
index 86c69f980927..7488e27c87c3 100644
--- a/mm/damon/sysfs.c
+++ b/mm/damon/sysfs.c
@@ -763,6 +763,7 @@ static const char * const damon_sysfs_damos_action_strs[] = {
"hugepage",
"nohugepage",
"lru_prio",
+ "lru_deprio",
"stat",
};
--
2.25.1
This commit adds a new DAMOS action called 'LRU_PRIO' for the physical
address space. The action prioritizes pages in the memory regions of
the user-specified target access pattern on their LRU lists. This is
hence supposed to be used for frequently accessed (hot) memory regions
so that hot pages could be more likely protected under memory pressure.
Internally, it simply calls 'mark_page_accessed()'.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/damon.h | 2 ++
mm/damon/ops-common.c | 42 ++++++++++++++++++++++++++++++++++++++++++
mm/damon/ops-common.h | 2 ++
mm/damon/paddr.c | 20 ++++++++++++++++++++
mm/damon/sysfs.c | 1 +
5 files changed, 67 insertions(+)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index b9aae19fab3e..4c64e03e94d8 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -86,6 +86,7 @@ struct damon_target {
* @DAMOS_PAGEOUT: Call ``madvise()`` for the region with MADV_PAGEOUT.
* @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
* @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
+ * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.
* @DAMOS_STAT: Do nothing but count the stat.
* @NR_DAMOS_ACTIONS: Total number of DAMOS actions
*/
@@ -95,6 +96,7 @@ enum damos_action {
DAMOS_PAGEOUT,
DAMOS_HUGEPAGE,
DAMOS_NOHUGEPAGE,
+ DAMOS_LRU_PRIO,
DAMOS_STAT, /* Do nothing but only record the stat */
NR_DAMOS_ACTIONS,
};
diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index 10ef20b2003f..b1335de200e7 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -130,3 +130,45 @@ int damon_pageout_score(struct damon_ctx *c, struct damon_region *r,
/* Return coldness of the region */
return DAMOS_MAX_SCORE - hotness;
}
+
+int damon_hot_score(struct damon_ctx *c, struct damon_region *r,
+ struct damos *s)
+{
+ unsigned int max_nr_accesses;
+ int freq_subscore;
+ unsigned int age_in_sec;
+ int age_in_log, age_subscore;
+ unsigned int freq_weight = s->quota.weight_nr_accesses;
+ unsigned int age_weight = s->quota.weight_age;
+ int hotness;
+
+ max_nr_accesses = c->aggr_interval / c->sample_interval;
+ freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE / max_nr_accesses;
+
+ age_in_sec = (unsigned long)r->age * c->aggr_interval / 1000000;
+ for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;
+ age_in_log++, age_in_sec >>= 1)
+ ;
+
+ /* If frequency is 0, higher age means it's colder */
+ if (freq_subscore == 0)
+ age_in_log *= -1;
+
+ /*
+ * Now age_in_log is in [-DAMON_MAX_AGE_IN_LOG, DAMON_MAX_AGE_IN_LOG].
+ * Scale it to be in [0, 100] and set it as age subscore.
+ */
+ age_in_log += DAMON_MAX_AGE_IN_LOG;
+ age_subscore = age_in_log * DAMON_MAX_SUBSCORE /
+ DAMON_MAX_AGE_IN_LOG / 2;
+
+ hotness = (freq_weight * freq_subscore + age_weight * age_subscore);
+ if (freq_weight + age_weight)
+ hotness /= freq_weight + age_weight;
+ /*
+ * Transform it to fit in [0, DAMOS_MAX_SCORE]
+ */
+ hotness = hotness * DAMOS_MAX_SCORE / DAMON_MAX_SUBSCORE;
+
+ return hotness;
+}
diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h
index e790cb5f8fe0..52329ff361cd 100644
--- a/mm/damon/ops-common.h
+++ b/mm/damon/ops-common.h
@@ -14,3 +14,5 @@ void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr);
int damon_pageout_score(struct damon_ctx *c, struct damon_region *r,
struct damos *s);
+int damon_hot_score(struct damon_ctx *c, struct damon_region *r,
+ struct damos *s);
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 7bcd48066b43..f145b1d51e13 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -233,6 +233,22 @@ static unsigned long damon_pa_pageout(struct damon_region *r)
return applied * PAGE_SIZE;
}
+static unsigned long damon_pa_mark_accessed(struct damon_region *r)
+{
+ unsigned long addr, applied = 0;
+
+ for (addr = r->ar.start; addr < r->ar.end; addr += PAGE_SIZE) {
+ struct page *page = damon_get_page(PHYS_PFN(addr));
+
+ if (!page)
+ continue;
+ mark_page_accessed(page);
+ put_page(page);
+ applied++;
+ }
+ return applied * PAGE_SIZE;
+}
+
static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx,
struct damon_target *t, struct damon_region *r,
struct damos *scheme)
@@ -240,6 +256,8 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx,
switch (scheme->action) {
case DAMOS_PAGEOUT:
return damon_pa_pageout(r);
+ case DAMOS_LRU_PRIO:
+ return damon_pa_mark_accessed(r);
default:
break;
}
@@ -253,6 +271,8 @@ static int damon_pa_scheme_score(struct damon_ctx *context,
switch (scheme->action) {
case DAMOS_PAGEOUT:
return damon_pageout_score(context, r, scheme);
+ case DAMOS_LRU_PRIO:
+ return damon_hot_score(context, r, scheme);
default:
break;
}
diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c
index c35809c6087c..86c69f980927 100644
--- a/mm/damon/sysfs.c
+++ b/mm/damon/sysfs.c
@@ -762,6 +762,7 @@ static const char * const damon_sysfs_damos_action_strs[] = {
"pageout",
"hugepage",
"nohugepage",
+ "lru_prio",
"stat",
};
--
2.25.1
On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
>
> Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> the hotness/coldness thresholds, CPU quota, and watermarks could be
> challenging for some users. To make the scheme easy to be used without
> complex tuning for common situations, this commit implements a static
> kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> 'LRU_DEPRIO' DAMOS actions.
>
> It proactively sorts LRU-lists using DAMON with conservatively chosen
> default values of the parameters. That is, the module under its default
> parameters will make no harm for common situations but provide some
> level of efficiency improvements for systems having clear hot/cold
> access pattern under a level of memory pressure while consuming only a
> limited small portion of CPU time.
Hi SeongJae,
While I believe DAMON pro-active reclamation and LRU-SORT
>
> Signed-off-by: SeongJae Park <[email protected]>
> ---
> mm/damon/Kconfig | 8 +
> mm/damon/Makefile | 1 +
> mm/damon/lru_sort.c | 546 ++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 555 insertions(+)
> create mode 100644 mm/damon/lru_sort.c
>
> diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
> index 9b559c76d6dd..66265e3a9c65 100644
> --- a/mm/damon/Kconfig
> +++ b/mm/damon/Kconfig
> @@ -92,4 +92,12 @@ config DAMON_RECLAIM
> reclamation under light memory pressure, while the traditional page
> scanning-based reclamation is used for heavy pressure.
>
> +config DAMON_LRU_SORT
> + bool "Build DAMON-based LRU-lists sorting (DAMON_LRU_SORT)"
> + depends on DAMON_PADDR
> + help
> + This builds the DAMON-based LRU-lists sorting subsystem. It tries to
> + protect frequently accessed (hot) pages while rarely accessed (cold)
> + pages reclaimed first under memory pressure.
> +
> endmenu
> diff --git a/mm/damon/Makefile b/mm/damon/Makefile
> index dbf7190b4144..3e6b8ad73858 100644
> --- a/mm/damon/Makefile
> +++ b/mm/damon/Makefile
> @@ -6,3 +6,4 @@ obj-$(CONFIG_DAMON_PADDR) += ops-common.o paddr.o
> obj-$(CONFIG_DAMON_SYSFS) += sysfs.o
> obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
> obj-$(CONFIG_DAMON_RECLAIM) += reclaim.o
> +obj-$(CONFIG_DAMON_LRU_SORT) += lru_sort.o
> diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c
> new file mode 100644
> index 000000000000..c276736a071c
> --- /dev/null
> +++ b/mm/damon/lru_sort.c
> @@ -0,0 +1,546 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * DAMON-based LRU-lists Sorting
> + *
> + * Author: SeongJae Park <[email protected]>
> + */
> +
> +#define pr_fmt(fmt) "damon-lru-sort: " fmt
> +
> +#include <linux/damon.h>
> +#include <linux/ioport.h>
> +#include <linux/module.h>
> +#include <linux/sched.h>
> +#include <linux/workqueue.h>
> +
> +#ifdef MODULE_PARAM_PREFIX
> +#undef MODULE_PARAM_PREFIX
> +#endif
> +#define MODULE_PARAM_PREFIX "damon_lru_sort."
> +
> +/*
> + * Enable or disable DAMON_LRU_SORT.
> + *
> + * You can enable DAMON_LRU_SORT by setting the value of this parameter as
> + * ``Y``. Setting it as ``N`` disables DAMON_LRU_SORT. Note that
> + * DAMON_LRU_SORT could do no real monitoring and LRU-lists sorting due to the
> + * watermarks-based activation condition. Refer to below descriptions for the
> + * watermarks parameter for this.
> + */
> +static bool enabled __read_mostly;
> +
> +/*
> + * Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``.
> + *
> + * Input parameters that updated while DAMON_LRU_SORT is running are not
> + * applied by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT
> + * reads values of parametrs except ``enabled`` again. Once the re-reading is
> + * done, this parameter is set as ``N``. If invalid parameters are found while
> + * the re-reading, DAMON_LRU_SORT will be disabled.
> + */
> +static bool commit_inputs __read_mostly;
> +module_param(commit_inputs, bool, 0600);
> +
> +/*
> + * Access frequency threshold for hot memory regions identification in permil.
> + *
> + * If a memory region is accessed in frequency of this or higher,
> + * DAMON_LRU_SORT identifies the region as hot, and mark it as accessed on the
> + * LRU list, so that it could not be reclaimed under memory pressure. 50% by
> + * default.
> + */
> +static unsigned long hot_thres_access_freq = 500;
> +module_param(hot_thres_access_freq, ulong, 0600);
> +
> +/*
> + * Time threshold for cold memory regions identification in microseconds.
> + *
> + * If a memory region is not accessed for this or longer time, DAMON_LRU_SORT
> + * identifies the region as cold, and mark it as unaccessed on the LRU list, so
> + * that it could be reclaimed first under memory pressure. 120 seconds by
> + * default.
> + */
> +static unsigned long cold_min_age __read_mostly = 120000000;
> +module_param(cold_min_age, ulong, 0600);
> +
> +/*
> + * Limit of time for trying the LRU lists sorting in milliseconds.
> + *
> + * DAMON_LRU_SORT tries to use only up to this time within a time window
> + * (quota_reset_interval_ms) for trying LRU lists sorting. This can be used
> + * for limiting CPU consumption of DAMON_LRU_SORT. If the value is zero, the
> + * limit is disabled.
> + *
> + * 10 ms by default.
> + */
> +static unsigned long quota_ms __read_mostly = 10;
> +module_param(quota_ms, ulong, 0600);
> +
> +/*
> + * The time quota charge reset interval in milliseconds.
> + *
> + * The charge reset interval for the quota of time (quota_ms). That is,
> + * DAMON_LRU_SORT does not try LRU-lists sorting for more than quota_ms
> + * milliseconds or quota_sz bytes within quota_reset_interval_ms milliseconds.
> + *
> + * 1 second by default.
> + */
> +static unsigned long quota_reset_interval_ms __read_mostly = 1000;
> +module_param(quota_reset_interval_ms, ulong, 0600);
> +
> +/*
> + * The watermarks check time interval in microseconds.
> + *
> + * Minimal time to wait before checking the watermarks, when DAMON_LRU_SORT is
> + * enabled but inactive due to its watermarks rule. 5 seconds by default.
> + */
> +static unsigned long wmarks_interval __read_mostly = 5000000;
> +module_param(wmarks_interval, ulong, 0600);
> +
> +/*
> + * Free memory rate (per thousand) for the high watermark.
> + *
> + * If free memory of the system in bytes per thousand bytes is higher than
> + * this, DAMON_LRU_SORT becomes inactive, so it does nothing but periodically
> + * checks the watermarks. 200 (20%) by default.
> + */
> +static unsigned long wmarks_high __read_mostly = 200;
> +module_param(wmarks_high, ulong, 0600);
> +
> +/*
> + * Free memory rate (per thousand) for the middle watermark.
> + *
> + * If free memory of the system in bytes per thousand bytes is between this and
> + * the low watermark, DAMON_LRU_SORT becomes active, so starts the monitoring
> + * and the LRU-lists sorting. 150 (15%) by default.
> + */
> +static unsigned long wmarks_mid __read_mostly = 150;
> +module_param(wmarks_mid, ulong, 0600);
> +
> +/*
> + * Free memory rate (per thousand) for the low watermark.
> + *
> + * If free memory of the system in bytes per thousand bytes is lower than this,
> + * DAMON_LRU_SORT becomes inactive, so it does nothing but periodically checks
> + * the watermarks. 50 (5%) by default.
> + */
> +static unsigned long wmarks_low __read_mostly = 50;
> +module_param(wmarks_low, ulong, 0600);
> +
> +/*
> + * Sampling interval for the monitoring in microseconds.
> + *
> + * The sampling interval of DAMON for the hot/cold memory monitoring. Please
> + * refer to the DAMON documentation for more detail. 5 ms by default.
> + */
> +static unsigned long sample_interval __read_mostly = 5000;
> +module_param(sample_interval, ulong, 0600);
> +
> +/*
> + * Aggregation interval for the monitoring in microseconds.
> + *
> + * The aggregation interval of DAMON for the hot/cold memory monitoring.
> + * Please refer to the DAMON documentation for more detail. 100 ms by default.
> + */
> +static unsigned long aggr_interval __read_mostly = 100000;
> +module_param(aggr_interval, ulong, 0600);
> +
> +/*
> + * Minimum number of monitoring regions.
> + *
> + * The minimal number of monitoring regions of DAMON for the hot/cold memory
> + * monitoring. This can be used to set lower-bound of the monitoring quality.
> + * But, setting this too high could result in increased monitoring overhead.
> + * Please refer to the DAMON documentation for more detail. 10 by default.
> + */
> +static unsigned long min_nr_regions __read_mostly = 10;
> +module_param(min_nr_regions, ulong, 0600);
> +
> +/*
> + * Maximum number of monitoring regions.
> + *
> + * The maximum number of monitoring regions of DAMON for the hot/cold memory
> + * monitoring. This can be used to set upper-bound of the monitoring overhead.
> + * However, setting this too low could result in bad monitoring quality.
> + * Please refer to the DAMON documentation for more detail. 1000 by default.
> + */
> +static unsigned long max_nr_regions __read_mostly = 1000;
> +module_param(max_nr_regions, ulong, 0600);
> +
> +/*
> + * Start of the target memory region in physical address.
> + *
> + * The start physical address of memory region that DAMON_LRU_SORT will do work
> + * against. By default, biggest System RAM is used as the region.
> + */
> +static unsigned long monitor_region_start __read_mostly;
> +module_param(monitor_region_start, ulong, 0600);
> +
> +/*
> + * End of the target memory region in physical address.
> + *
> + * The end physical address of memory region that DAMON_LRU_SORT will do work
> + * against. By default, biggest System RAM is used as the region.
> + */
> +static unsigned long monitor_region_end __read_mostly;
> +module_param(monitor_region_end, ulong, 0600);
> +
> +/*
> + * PID of the DAMON thread
> + *
> + * If DAMON_LRU_SORT is enabled, this becomes the PID of the worker thread.
> + * Else, -1.
> + */
> +static int kdamond_pid __read_mostly = -1;
> +module_param(kdamond_pid, int, 0400);
> +
> +/*
> + * Number of hot memory regions that tried to be LRU-sorted.
> + */
> +static unsigned long nr_lru_sort_tried_hot_regions __read_mostly;
> +module_param(nr_lru_sort_tried_hot_regions, ulong, 0400);
> +
> +/*
> + * Total bytes of hot memory regions that tried to be LRU-sorted.
> + */
> +static unsigned long bytes_lru_sort_tried_hot_regions __read_mostly;
> +module_param(bytes_lru_sort_tried_hot_regions, ulong, 0400);
> +
> +/*
> + * Number of hot memory regions that successfully be LRU-sorted.
> + */
> +static unsigned long nr_lru_sorted_hot_regions __read_mostly;
> +module_param(nr_lru_sorted_hot_regions, ulong, 0400);
> +
> +/*
> + * Total bytes of hot memory regions that successfully be LRU-sorted.
> + */
> +static unsigned long bytes_lru_sorted_hot_regions __read_mostly;
> +module_param(bytes_lru_sorted_hot_regions, ulong, 0400);
> +
> +/*
> + * Number of times that the time quota limit for hot regions have exceeded
> + */
> +static unsigned long nr_hot_quota_exceeds __read_mostly;
> +module_param(nr_hot_quota_exceeds, ulong, 0400);
> +
> +/*
> + * Number of cold memory regions that tried to be LRU-sorted.
> + */
> +static unsigned long nr_lru_sort_tried_cold_regions __read_mostly;
> +module_param(nr_lru_sort_tried_cold_regions, ulong, 0400);
> +
> +/*
> + * Total bytes of cold memory regions that tried to be LRU-sorted.
> + */
> +static unsigned long bytes_lru_sort_tried_cold_regions __read_mostly;
> +module_param(bytes_lru_sort_tried_cold_regions, ulong, 0400);
> +
> +/*
> + * Number of cold memory regions that successfully be LRU-sorted.
> + */
> +static unsigned long nr_lru_sorted_cold_regions __read_mostly;
> +module_param(nr_lru_sorted_cold_regions, ulong, 0400);
> +
> +/*
> + * Total bytes of cold memory regions that successfully be LRU-sorted.
> + */
> +static unsigned long bytes_lru_sorted_cold_regions __read_mostly;
> +module_param(bytes_lru_sorted_cold_regions, ulong, 0400);
> +
> +/*
> + * Number of times that the time quota limit for cold regions have exceeded
> + */
> +static unsigned long nr_cold_quota_exceeds __read_mostly;
> +module_param(nr_cold_quota_exceeds, ulong, 0400);
> +
> +static struct damon_ctx *ctx;
> +static struct damon_target *target;
> +
> +struct damon_lru_sort_ram_walk_arg {
> + unsigned long start;
> + unsigned long end;
> +};
> +
> +static int walk_system_ram(struct resource *res, void *arg)
> +{
> + struct damon_lru_sort_ram_walk_arg *a = arg;
> +
> + if (a->end - a->start < resource_size(res)) {
> + a->start = res->start;
> + a->end = res->end;
> + }
> + return 0;
> +}
> +
> +/*
> + * Find biggest 'System RAM' resource and store its start and end address in
> + * @start and @end, respectively. If no System RAM is found, returns false.
> + */
> +static bool get_monitoring_region(unsigned long *start, unsigned long *end)
> +{
> + struct damon_lru_sort_ram_walk_arg arg = {};
> +
> + walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram);
> + if (arg.end <= arg.start)
> + return false;
> +
> + *start = arg.start;
> + *end = arg.end;
> + return true;
> +}
> +
> +/* Create a DAMON-based operation scheme for hot memory regions */
> +static struct damos *damon_lru_sort_new_hot_scheme(unsigned int hot_thres)
> +{
> + struct damos_watermarks wmarks = {
> + .metric = DAMOS_WMARK_FREE_MEM_RATE,
> + .interval = wmarks_interval,
> + .high = wmarks_high,
> + .mid = wmarks_mid,
> + .low = wmarks_low,
> + };
> + struct damos_quota quota = {
> + /*
> + * Do not try LRU-lists sorting of hot pages for more than half
> + * of quota_ms milliseconds within quota_reset_interval_ms.
> + */
> + .ms = quota_ms / 2,
> + .sz = 0,
> + .reset_interval = quota_reset_interval_ms,
> + /* Within the quota, mark hotter regions accessed first. */
> + .weight_sz = 0,
> + .weight_nr_accesses = 1,
> + .weight_age = 0,
> + };
> + struct damos *scheme = damon_new_scheme(
> + /* Find regions having PAGE_SIZE or larger size */
> + PAGE_SIZE, ULONG_MAX,
> + /* and accessed for more than the threshold */
> + hot_thres, UINT_MAX,
> + /* no matter its age */
> + 0, UINT_MAX,
> + /* prioritize those on LRU lists, as soon as found */
> + DAMOS_LRU_PRIO,
> + /* under the quota. */
> + "a,
> + /* (De)activate this according to the watermarks. */
> + &wmarks);
> +
> + return scheme;
> +}
> +
> +/* Create a DAMON-based operation scheme for cold memory regions */
> +static struct damos *damon_lru_sort_new_cold_scheme(unsigned int cold_thres)
> +{
> + struct damos_watermarks wmarks = {
> + .metric = DAMOS_WMARK_FREE_MEM_RATE,
> + .interval = wmarks_interval,
> + .high = wmarks_high,
> + .mid = wmarks_mid,
> + .low = wmarks_low,
> + };
> + struct damos_quota quota = {
> + /*
> + * Do not try LRU-lists sorting of cold pages for more than
> + * half of quota_ms milliseconds within
> + * quota_reset_interval_ms.
> + */
> + .ms = quota_ms / 2,
> + .sz = 0,
> + .reset_interval = quota_reset_interval_ms,
> + /* Within the quota, mark colder regions not accessed first. */
> + .weight_sz = 0,
> + .weight_nr_accesses = 0,
> + .weight_age = 1,
> + };
> + struct damos *scheme = damon_new_scheme(
> + /* Find regions having PAGE_SIZE or larger size */
> + PAGE_SIZE, ULONG_MAX,
> + /* and not accessed at all */
> + 0, 0,
> + /* for cold_thres or more micro-seconds, and */
> + cold_thres, UINT_MAX,
> + /* mark those as not accessed, as soon as found */
> + DAMOS_LRU_DEPRIO,
> + /* under the quota. */
> + "a,
> + /* (De)activate this according to the watermarks. */
> + &wmarks);
> +
> + return scheme;
> +}
> +
> +static int damon_lru_sort_apply_parameters(void)
> +{
> + struct damos *scheme, *next_scheme;
> + struct damon_addr_range addr_range;
> + unsigned int hot_thres, cold_thres;
> + int err = 0;
> +
> + err = damon_set_attrs(ctx, sample_interval, aggr_interval, 0,
> + min_nr_regions, max_nr_regions);
> + if (err)
> + return err;
> +
> + /* free previously set schemes */
> + damon_for_each_scheme_safe(scheme, next_scheme, ctx)
> + damon_destroy_scheme(scheme);
> +
> + /* aggr_interval / sample_interval is the maximum nr_accesses */
> + hot_thres = aggr_interval / sample_interval * hot_thres_access_freq /
> + 1000;
> + scheme = damon_lru_sort_new_hot_scheme(hot_thres);
> + if (!scheme)
> + return -ENOMEM;
> + damon_add_scheme(ctx, scheme);
> +
> + cold_thres = cold_min_age / aggr_interval;
> + scheme = damon_lru_sort_new_cold_scheme(cold_thres);
> + if (!scheme)
> + return -ENOMEM;
> + damon_add_scheme(ctx, scheme);
> +
> + if (monitor_region_start > monitor_region_end)
> + return -EINVAL;
> + if (!monitor_region_start && !monitor_region_end &&
> + !get_monitoring_region(&monitor_region_start,
> + &monitor_region_end))
> + return -EINVAL;
> + addr_range.start = monitor_region_start;
> + addr_range.end = monitor_region_end;
> + return damon_set_regions(target, &addr_range, 1);
> +}
> +
> +static int damon_lru_sort_turn(bool on)
> +{
> + int err;
> +
> + if (!on) {
> + err = damon_stop(&ctx, 1);
> + if (!err)
> + kdamond_pid = -1;
> + return err;
> + }
> +
> + err = damon_lru_sort_apply_parameters();
> + if (err)
> + return err;
> +
> + err = damon_start(&ctx, 1, true);
> + if (err)
> + return err;
> + kdamond_pid = ctx->kdamond->pid;
> + return 0;
> +}
> +
> +static struct delayed_work damon_lru_sort_timer;
> +static void damon_lru_sort_timer_fn(struct work_struct *work)
> +{
> + static bool last_enabled;
> + bool now_enabled;
> +
> + now_enabled = enabled;
> + if (last_enabled != now_enabled) {
> + if (!damon_lru_sort_turn(now_enabled))
> + last_enabled = now_enabled;
> + else
> + enabled = last_enabled;
> + }
> +}
> +static DECLARE_DELAYED_WORK(damon_lru_sort_timer, damon_lru_sort_timer_fn);
> +
> +static bool damon_lru_sort_initialized;
> +
> +static int damon_lru_sort_enabled_store(const char *val,
> + const struct kernel_param *kp)
> +{
> + int rc = param_set_bool(val, kp);
> +
> + if (rc < 0)
> + return rc;
> +
> + if (!damon_lru_sort_initialized)
> + return rc;
> +
> + schedule_delayed_work(&damon_lru_sort_timer, 0);
> +
> + return 0;
> +}
> +
> +static const struct kernel_param_ops enabled_param_ops = {
> + .set = damon_lru_sort_enabled_store,
> + .get = param_get_bool,
> +};
> +
> +module_param_cb(enabled, &enabled_param_ops, &enabled, 0600);
> +MODULE_PARM_DESC(enabled,
> + "Enable or disable DAMON_LRU_SORT (default: disabled)");
> +
> +static int damon_lru_sort_handle_commit_inputs(void)
> +{
> + int err;
> +
> + if (!commit_inputs)
> + return 0;
> +
> + err = damon_lru_sort_apply_parameters();
> + commit_inputs = false;
> + return err;
> +}
> +
> +static int damon_lru_sort_after_aggregation(struct damon_ctx *c)
> +{
> + struct damos *s;
> +
> + /* update the stats parameter */
> + damon_for_each_scheme(s, c) {
> + if (s->action == DAMOS_LRU_PRIO) {
> + nr_lru_sort_tried_hot_regions = s->stat.nr_tried;
> + bytes_lru_sort_tried_hot_regions = s->stat.sz_tried;
> + nr_lru_sorted_hot_regions = s->stat.nr_applied;
> + bytes_lru_sorted_hot_regions = s->stat.sz_applied;
> + nr_hot_quota_exceeds = s->stat.qt_exceeds;
> + } else if (s->action == DAMOS_LRU_DEPRIO) {
> + nr_lru_sort_tried_cold_regions = s->stat.nr_tried;
> + bytes_lru_sort_tried_cold_regions = s->stat.sz_tried;
> + nr_lru_sorted_cold_regions = s->stat.nr_applied;
> + bytes_lru_sorted_cold_regions = s->stat.sz_applied;
> + nr_cold_quota_exceeds = s->stat.qt_exceeds;
> + }
> + }
> +
> + return damon_lru_sort_handle_commit_inputs();
> +}
> +
> +static int damon_lru_sort_after_wmarks_check(struct damon_ctx *c)
> +{
> + return damon_lru_sort_handle_commit_inputs();
> +}
> +
> +static int __init damon_lru_sort_init(void)
> +{
> + ctx = damon_new_ctx();
> + if (!ctx)
> + return -ENOMEM;
> +
> + if (damon_select_ops(ctx, DAMON_OPS_PADDR))
> + return -EINVAL;
> +
> + ctx->callback.after_wmarks_check = damon_lru_sort_after_wmarks_check;
> + ctx->callback.after_aggregation = damon_lru_sort_after_aggregation;
> +
> + target = damon_new_target();
> + if (!target) {
> + damon_destroy_ctx(ctx);
> + return -ENOMEM;
> + }
> + damon_add_target(ctx, target);
> +
> + schedule_delayed_work(&damon_lru_sort_timer, 0);
> +
> + damon_lru_sort_initialized = true;
> + return 0;
> +}
> +
> +module_init(damon_lru_sort_init);
> --
> 2.25.1
>
On Thu, Sep 1, 2022 at 2:03 PM Barry Song <[email protected]> wrote:
>
> On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
> >
> > Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> > 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> > the hotness/coldness thresholds, CPU quota, and watermarks could be
> > challenging for some users. To make the scheme easy to be used without
> > complex tuning for common situations, this commit implements a static
> > kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> > 'LRU_DEPRIO' DAMOS actions.
> >
> > It proactively sorts LRU-lists using DAMON with conservatively chosen
> > default values of the parameters. That is, the module under its default
> > parameters will make no harm for common situations but provide some
> > level of efficiency improvements for systems having clear hot/cold
> > access pattern under a level of memory pressure while consuming only a
> > limited small portion of CPU time.
>
Hi SeongJae,
While I believe DAMON pro-active reclamation and LRU-SORT can benefit the system
by either swapping out cold pages earlier and sorting LRU lists before
system has high
memory pressure, I am still not convinced the improvement really comes from the
identification of cold and hot pages by DAMON.
My guess is that even if we randomly pick some regions in memory and do the same
thing in the kernel, we can also see the improvement.
As we actually depend on two facts to benefit from DAMON
1. locality
while virtual address might have some locality, physical address seems
not. for example,
address A might be mapped by facebook, address A + 4096 could be
mapped by youtube.
There is nothing which can stop contiguous physical addresses from
being mapped by
completely irrelevant applications. so regions based on paddr seems pointless.
2. accuration
As I have reported it is very hard for damon to accurately track
virtual address since
virtual space is so huge:
https://lore.kernel.org/lkml/CAGsJ_4x_k9009HwpTswEq1ut_co8XYdpZ9k0BVW=0=HRiifxkA@mail.gmail.com/
I believe it is also true for paddr since paddr has much worse
locality than vaddr.
so we probably need a lot of regions, ideally, one region for each page.
To me, it seems neither of these two facts are true. So I am more
willing to believe
that the benefits come from areas picked randomly.
Am I missing something?
>
> >
> > Signed-off-by: SeongJae Park <[email protected]>
> > ---
> > mm/damon/Kconfig | 8 +
> > mm/damon/Makefile | 1 +
> > mm/damon/lru_sort.c | 546 ++++++++++++++++++++++++++++++++++++++++++++
> > 3 files changed, 555 insertions(+)
> > create mode 100644 mm/damon/lru_sort.c
> >
> > diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
> > index 9b559c76d6dd..66265e3a9c65 100644
> > --- a/mm/damon/Kconfig
> > +++ b/mm/damon/Kconfig
> > @@ -92,4 +92,12 @@ config DAMON_RECLAIM
> > reclamation under light memory pressure, while the traditional page
> > scanning-based reclamation is used for heavy pressure.
> >
> > +config DAMON_LRU_SORT
> > + bool "Build DAMON-based LRU-lists sorting (DAMON_LRU_SORT)"
> > + depends on DAMON_PADDR
> > + help
> > + This builds the DAMON-based LRU-lists sorting subsystem. It tries to
> > + protect frequently accessed (hot) pages while rarely accessed (cold)
> > + pages reclaimed first under memory pressure.
> > +
> > endmenu
> > diff --git a/mm/damon/Makefile b/mm/damon/Makefile
> > index dbf7190b4144..3e6b8ad73858 100644
> > --- a/mm/damon/Makefile
> > +++ b/mm/damon/Makefile
> > @@ -6,3 +6,4 @@ obj-$(CONFIG_DAMON_PADDR) += ops-common.o paddr.o
> > obj-$(CONFIG_DAMON_SYSFS) += sysfs.o
> > obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
> > obj-$(CONFIG_DAMON_RECLAIM) += reclaim.o
> > +obj-$(CONFIG_DAMON_LRU_SORT) += lru_sort.o
> > diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c
> > new file mode 100644
> > index 000000000000..c276736a071c
> > --- /dev/null
> > +++ b/mm/damon/lru_sort.c
> > @@ -0,0 +1,546 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * DAMON-based LRU-lists Sorting
> > + *
> > + * Author: SeongJae Park <[email protected]>
> > + */
> > +
> > +#define pr_fmt(fmt) "damon-lru-sort: " fmt
> > +
> > +#include <linux/damon.h>
> > +#include <linux/ioport.h>
> > +#include <linux/module.h>
> > +#include <linux/sched.h>
> > +#include <linux/workqueue.h>
> > +
> > +#ifdef MODULE_PARAM_PREFIX
> > +#undef MODULE_PARAM_PREFIX
> > +#endif
> > +#define MODULE_PARAM_PREFIX "damon_lru_sort."
> > +
> > +/*
> > + * Enable or disable DAMON_LRU_SORT.
> > + *
> > + * You can enable DAMON_LRU_SORT by setting the value of this parameter as
> > + * ``Y``. Setting it as ``N`` disables DAMON_LRU_SORT. Note that
> > + * DAMON_LRU_SORT could do no real monitoring and LRU-lists sorting due to the
> > + * watermarks-based activation condition. Refer to below descriptions for the
> > + * watermarks parameter for this.
> > + */
> > +static bool enabled __read_mostly;
> > +
> > +/*
> > + * Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``.
> > + *
> > + * Input parameters that updated while DAMON_LRU_SORT is running are not
> > + * applied by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT
> > + * reads values of parametrs except ``enabled`` again. Once the re-reading is
> > + * done, this parameter is set as ``N``. If invalid parameters are found while
> > + * the re-reading, DAMON_LRU_SORT will be disabled.
> > + */
> > +static bool commit_inputs __read_mostly;
> > +module_param(commit_inputs, bool, 0600);
> > +
> > +/*
> > + * Access frequency threshold for hot memory regions identification in permil.
> > + *
> > + * If a memory region is accessed in frequency of this or higher,
> > + * DAMON_LRU_SORT identifies the region as hot, and mark it as accessed on the
> > + * LRU list, so that it could not be reclaimed under memory pressure. 50% by
> > + * default.
> > + */
> > +static unsigned long hot_thres_access_freq = 500;
> > +module_param(hot_thres_access_freq, ulong, 0600);
> > +
> > +/*
> > + * Time threshold for cold memory regions identification in microseconds.
> > + *
> > + * If a memory region is not accessed for this or longer time, DAMON_LRU_SORT
> > + * identifies the region as cold, and mark it as unaccessed on the LRU list, so
> > + * that it could be reclaimed first under memory pressure. 120 seconds by
> > + * default.
> > + */
> > +static unsigned long cold_min_age __read_mostly = 120000000;
> > +module_param(cold_min_age, ulong, 0600);
> > +
> > +/*
> > + * Limit of time for trying the LRU lists sorting in milliseconds.
> > + *
> > + * DAMON_LRU_SORT tries to use only up to this time within a time window
> > + * (quota_reset_interval_ms) for trying LRU lists sorting. This can be used
> > + * for limiting CPU consumption of DAMON_LRU_SORT. If the value is zero, the
> > + * limit is disabled.
> > + *
> > + * 10 ms by default.
> > + */
> > +static unsigned long quota_ms __read_mostly = 10;
> > +module_param(quota_ms, ulong, 0600);
> > +
> > +/*
> > + * The time quota charge reset interval in milliseconds.
> > + *
> > + * The charge reset interval for the quota of time (quota_ms). That is,
> > + * DAMON_LRU_SORT does not try LRU-lists sorting for more than quota_ms
> > + * milliseconds or quota_sz bytes within quota_reset_interval_ms milliseconds.
> > + *
> > + * 1 second by default.
> > + */
> > +static unsigned long quota_reset_interval_ms __read_mostly = 1000;
> > +module_param(quota_reset_interval_ms, ulong, 0600);
> > +
> > +/*
> > + * The watermarks check time interval in microseconds.
> > + *
> > + * Minimal time to wait before checking the watermarks, when DAMON_LRU_SORT is
> > + * enabled but inactive due to its watermarks rule. 5 seconds by default.
> > + */
> > +static unsigned long wmarks_interval __read_mostly = 5000000;
> > +module_param(wmarks_interval, ulong, 0600);
> > +
> > +/*
> > + * Free memory rate (per thousand) for the high watermark.
> > + *
> > + * If free memory of the system in bytes per thousand bytes is higher than
> > + * this, DAMON_LRU_SORT becomes inactive, so it does nothing but periodically
> > + * checks the watermarks. 200 (20%) by default.
> > + */
> > +static unsigned long wmarks_high __read_mostly = 200;
> > +module_param(wmarks_high, ulong, 0600);
> > +
> > +/*
> > + * Free memory rate (per thousand) for the middle watermark.
> > + *
> > + * If free memory of the system in bytes per thousand bytes is between this and
> > + * the low watermark, DAMON_LRU_SORT becomes active, so starts the monitoring
> > + * and the LRU-lists sorting. 150 (15%) by default.
> > + */
> > +static unsigned long wmarks_mid __read_mostly = 150;
> > +module_param(wmarks_mid, ulong, 0600);
> > +
> > +/*
> > + * Free memory rate (per thousand) for the low watermark.
> > + *
> > + * If free memory of the system in bytes per thousand bytes is lower than this,
> > + * DAMON_LRU_SORT becomes inactive, so it does nothing but periodically checks
> > + * the watermarks. 50 (5%) by default.
> > + */
> > +static unsigned long wmarks_low __read_mostly = 50;
> > +module_param(wmarks_low, ulong, 0600);
> > +
> > +/*
> > + * Sampling interval for the monitoring in microseconds.
> > + *
> > + * The sampling interval of DAMON for the hot/cold memory monitoring. Please
> > + * refer to the DAMON documentation for more detail. 5 ms by default.
> > + */
> > +static unsigned long sample_interval __read_mostly = 5000;
> > +module_param(sample_interval, ulong, 0600);
> > +
> > +/*
> > + * Aggregation interval for the monitoring in microseconds.
> > + *
> > + * The aggregation interval of DAMON for the hot/cold memory monitoring.
> > + * Please refer to the DAMON documentation for more detail. 100 ms by default.
> > + */
> > +static unsigned long aggr_interval __read_mostly = 100000;
> > +module_param(aggr_interval, ulong, 0600);
> > +
> > +/*
> > + * Minimum number of monitoring regions.
> > + *
> > + * The minimal number of monitoring regions of DAMON for the hot/cold memory
> > + * monitoring. This can be used to set lower-bound of the monitoring quality.
> > + * But, setting this too high could result in increased monitoring overhead.
> > + * Please refer to the DAMON documentation for more detail. 10 by default.
> > + */
> > +static unsigned long min_nr_regions __read_mostly = 10;
> > +module_param(min_nr_regions, ulong, 0600);
> > +
> > +/*
> > + * Maximum number of monitoring regions.
> > + *
> > + * The maximum number of monitoring regions of DAMON for the hot/cold memory
> > + * monitoring. This can be used to set upper-bound of the monitoring overhead.
> > + * However, setting this too low could result in bad monitoring quality.
> > + * Please refer to the DAMON documentation for more detail. 1000 by default.
> > + */
> > +static unsigned long max_nr_regions __read_mostly = 1000;
> > +module_param(max_nr_regions, ulong, 0600);
> > +
> > +/*
> > + * Start of the target memory region in physical address.
> > + *
> > + * The start physical address of memory region that DAMON_LRU_SORT will do work
> > + * against. By default, biggest System RAM is used as the region.
> > + */
> > +static unsigned long monitor_region_start __read_mostly;
> > +module_param(monitor_region_start, ulong, 0600);
> > +
> > +/*
> > + * End of the target memory region in physical address.
> > + *
> > + * The end physical address of memory region that DAMON_LRU_SORT will do work
> > + * against. By default, biggest System RAM is used as the region.
> > + */
> > +static unsigned long monitor_region_end __read_mostly;
> > +module_param(monitor_region_end, ulong, 0600);
> > +
> > +/*
> > + * PID of the DAMON thread
> > + *
> > + * If DAMON_LRU_SORT is enabled, this becomes the PID of the worker thread.
> > + * Else, -1.
> > + */
> > +static int kdamond_pid __read_mostly = -1;
> > +module_param(kdamond_pid, int, 0400);
> > +
> > +/*
> > + * Number of hot memory regions that tried to be LRU-sorted.
> > + */
> > +static unsigned long nr_lru_sort_tried_hot_regions __read_mostly;
> > +module_param(nr_lru_sort_tried_hot_regions, ulong, 0400);
> > +
> > +/*
> > + * Total bytes of hot memory regions that tried to be LRU-sorted.
> > + */
> > +static unsigned long bytes_lru_sort_tried_hot_regions __read_mostly;
> > +module_param(bytes_lru_sort_tried_hot_regions, ulong, 0400);
> > +
> > +/*
> > + * Number of hot memory regions that successfully be LRU-sorted.
> > + */
> > +static unsigned long nr_lru_sorted_hot_regions __read_mostly;
> > +module_param(nr_lru_sorted_hot_regions, ulong, 0400);
> > +
> > +/*
> > + * Total bytes of hot memory regions that successfully be LRU-sorted.
> > + */
> > +static unsigned long bytes_lru_sorted_hot_regions __read_mostly;
> > +module_param(bytes_lru_sorted_hot_regions, ulong, 0400);
> > +
> > +/*
> > + * Number of times that the time quota limit for hot regions have exceeded
> > + */
> > +static unsigned long nr_hot_quota_exceeds __read_mostly;
> > +module_param(nr_hot_quota_exceeds, ulong, 0400);
> > +
> > +/*
> > + * Number of cold memory regions that tried to be LRU-sorted.
> > + */
> > +static unsigned long nr_lru_sort_tried_cold_regions __read_mostly;
> > +module_param(nr_lru_sort_tried_cold_regions, ulong, 0400);
> > +
> > +/*
> > + * Total bytes of cold memory regions that tried to be LRU-sorted.
> > + */
> > +static unsigned long bytes_lru_sort_tried_cold_regions __read_mostly;
> > +module_param(bytes_lru_sort_tried_cold_regions, ulong, 0400);
> > +
> > +/*
> > + * Number of cold memory regions that successfully be LRU-sorted.
> > + */
> > +static unsigned long nr_lru_sorted_cold_regions __read_mostly;
> > +module_param(nr_lru_sorted_cold_regions, ulong, 0400);
> > +
> > +/*
> > + * Total bytes of cold memory regions that successfully be LRU-sorted.
> > + */
> > +static unsigned long bytes_lru_sorted_cold_regions __read_mostly;
> > +module_param(bytes_lru_sorted_cold_regions, ulong, 0400);
> > +
> > +/*
> > + * Number of times that the time quota limit for cold regions have exceeded
> > + */
> > +static unsigned long nr_cold_quota_exceeds __read_mostly;
> > +module_param(nr_cold_quota_exceeds, ulong, 0400);
> > +
> > +static struct damon_ctx *ctx;
> > +static struct damon_target *target;
> > +
> > +struct damon_lru_sort_ram_walk_arg {
> > + unsigned long start;
> > + unsigned long end;
> > +};
> > +
> > +static int walk_system_ram(struct resource *res, void *arg)
> > +{
> > + struct damon_lru_sort_ram_walk_arg *a = arg;
> > +
> > + if (a->end - a->start < resource_size(res)) {
> > + a->start = res->start;
> > + a->end = res->end;
> > + }
> > + return 0;
> > +}
> > +
> > +/*
> > + * Find biggest 'System RAM' resource and store its start and end address in
> > + * @start and @end, respectively. If no System RAM is found, returns false.
> > + */
> > +static bool get_monitoring_region(unsigned long *start, unsigned long *end)
> > +{
> > + struct damon_lru_sort_ram_walk_arg arg = {};
> > +
> > + walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram);
> > + if (arg.end <= arg.start)
> > + return false;
> > +
> > + *start = arg.start;
> > + *end = arg.end;
> > + return true;
> > +}
> > +
> > +/* Create a DAMON-based operation scheme for hot memory regions */
> > +static struct damos *damon_lru_sort_new_hot_scheme(unsigned int hot_thres)
> > +{
> > + struct damos_watermarks wmarks = {
> > + .metric = DAMOS_WMARK_FREE_MEM_RATE,
> > + .interval = wmarks_interval,
> > + .high = wmarks_high,
> > + .mid = wmarks_mid,
> > + .low = wmarks_low,
> > + };
> > + struct damos_quota quota = {
> > + /*
> > + * Do not try LRU-lists sorting of hot pages for more than half
> > + * of quota_ms milliseconds within quota_reset_interval_ms.
> > + */
> > + .ms = quota_ms / 2,
> > + .sz = 0,
> > + .reset_interval = quota_reset_interval_ms,
> > + /* Within the quota, mark hotter regions accessed first. */
> > + .weight_sz = 0,
> > + .weight_nr_accesses = 1,
> > + .weight_age = 0,
> > + };
> > + struct damos *scheme = damon_new_scheme(
> > + /* Find regions having PAGE_SIZE or larger size */
> > + PAGE_SIZE, ULONG_MAX,
> > + /* and accessed for more than the threshold */
> > + hot_thres, UINT_MAX,
> > + /* no matter its age */
> > + 0, UINT_MAX,
> > + /* prioritize those on LRU lists, as soon as found */
> > + DAMOS_LRU_PRIO,
> > + /* under the quota. */
> > + "a,
> > + /* (De)activate this according to the watermarks. */
> > + &wmarks);
> > +
> > + return scheme;
> > +}
> > +
> > +/* Create a DAMON-based operation scheme for cold memory regions */
> > +static struct damos *damon_lru_sort_new_cold_scheme(unsigned int cold_thres)
> > +{
> > + struct damos_watermarks wmarks = {
> > + .metric = DAMOS_WMARK_FREE_MEM_RATE,
> > + .interval = wmarks_interval,
> > + .high = wmarks_high,
> > + .mid = wmarks_mid,
> > + .low = wmarks_low,
> > + };
> > + struct damos_quota quota = {
> > + /*
> > + * Do not try LRU-lists sorting of cold pages for more than
> > + * half of quota_ms milliseconds within
> > + * quota_reset_interval_ms.
> > + */
> > + .ms = quota_ms / 2,
> > + .sz = 0,
> > + .reset_interval = quota_reset_interval_ms,
> > + /* Within the quota, mark colder regions not accessed first. */
> > + .weight_sz = 0,
> > + .weight_nr_accesses = 0,
> > + .weight_age = 1,
> > + };
> > + struct damos *scheme = damon_new_scheme(
> > + /* Find regions having PAGE_SIZE or larger size */
> > + PAGE_SIZE, ULONG_MAX,
> > + /* and not accessed at all */
> > + 0, 0,
> > + /* for cold_thres or more micro-seconds, and */
> > + cold_thres, UINT_MAX,
> > + /* mark those as not accessed, as soon as found */
> > + DAMOS_LRU_DEPRIO,
> > + /* under the quota. */
> > + "a,
> > + /* (De)activate this according to the watermarks. */
> > + &wmarks);
> > +
> > + return scheme;
> > +}
> > +
> > +static int damon_lru_sort_apply_parameters(void)
> > +{
> > + struct damos *scheme, *next_scheme;
> > + struct damon_addr_range addr_range;
> > + unsigned int hot_thres, cold_thres;
> > + int err = 0;
> > +
> > + err = damon_set_attrs(ctx, sample_interval, aggr_interval, 0,
> > + min_nr_regions, max_nr_regions);
> > + if (err)
> > + return err;
> > +
> > + /* free previously set schemes */
> > + damon_for_each_scheme_safe(scheme, next_scheme, ctx)
> > + damon_destroy_scheme(scheme);
> > +
> > + /* aggr_interval / sample_interval is the maximum nr_accesses */
> > + hot_thres = aggr_interval / sample_interval * hot_thres_access_freq /
> > + 1000;
> > + scheme = damon_lru_sort_new_hot_scheme(hot_thres);
> > + if (!scheme)
> > + return -ENOMEM;
> > + damon_add_scheme(ctx, scheme);
> > +
> > + cold_thres = cold_min_age / aggr_interval;
> > + scheme = damon_lru_sort_new_cold_scheme(cold_thres);
> > + if (!scheme)
> > + return -ENOMEM;
> > + damon_add_scheme(ctx, scheme);
> > +
> > + if (monitor_region_start > monitor_region_end)
> > + return -EINVAL;
> > + if (!monitor_region_start && !monitor_region_end &&
> > + !get_monitoring_region(&monitor_region_start,
> > + &monitor_region_end))
> > + return -EINVAL;
> > + addr_range.start = monitor_region_start;
> > + addr_range.end = monitor_region_end;
> > + return damon_set_regions(target, &addr_range, 1);
> > +}
> > +
> > +static int damon_lru_sort_turn(bool on)
> > +{
> > + int err;
> > +
> > + if (!on) {
> > + err = damon_stop(&ctx, 1);
> > + if (!err)
> > + kdamond_pid = -1;
> > + return err;
> > + }
> > +
> > + err = damon_lru_sort_apply_parameters();
> > + if (err)
> > + return err;
> > +
> > + err = damon_start(&ctx, 1, true);
> > + if (err)
> > + return err;
> > + kdamond_pid = ctx->kdamond->pid;
> > + return 0;
> > +}
> > +
> > +static struct delayed_work damon_lru_sort_timer;
> > +static void damon_lru_sort_timer_fn(struct work_struct *work)
> > +{
> > + static bool last_enabled;
> > + bool now_enabled;
> > +
> > + now_enabled = enabled;
> > + if (last_enabled != now_enabled) {
> > + if (!damon_lru_sort_turn(now_enabled))
> > + last_enabled = now_enabled;
> > + else
> > + enabled = last_enabled;
> > + }
> > +}
> > +static DECLARE_DELAYED_WORK(damon_lru_sort_timer, damon_lru_sort_timer_fn);
> > +
> > +static bool damon_lru_sort_initialized;
> > +
> > +static int damon_lru_sort_enabled_store(const char *val,
> > + const struct kernel_param *kp)
> > +{
> > + int rc = param_set_bool(val, kp);
> > +
> > + if (rc < 0)
> > + return rc;
> > +
> > + if (!damon_lru_sort_initialized)
> > + return rc;
> > +
> > + schedule_delayed_work(&damon_lru_sort_timer, 0);
> > +
> > + return 0;
> > +}
> > +
> > +static const struct kernel_param_ops enabled_param_ops = {
> > + .set = damon_lru_sort_enabled_store,
> > + .get = param_get_bool,
> > +};
> > +
> > +module_param_cb(enabled, &enabled_param_ops, &enabled, 0600);
> > +MODULE_PARM_DESC(enabled,
> > + "Enable or disable DAMON_LRU_SORT (default: disabled)");
> > +
> > +static int damon_lru_sort_handle_commit_inputs(void)
> > +{
> > + int err;
> > +
> > + if (!commit_inputs)
> > + return 0;
> > +
> > + err = damon_lru_sort_apply_parameters();
> > + commit_inputs = false;
> > + return err;
> > +}
> > +
> > +static int damon_lru_sort_after_aggregation(struct damon_ctx *c)
> > +{
> > + struct damos *s;
> > +
> > + /* update the stats parameter */
> > + damon_for_each_scheme(s, c) {
> > + if (s->action == DAMOS_LRU_PRIO) {
> > + nr_lru_sort_tried_hot_regions = s->stat.nr_tried;
> > + bytes_lru_sort_tried_hot_regions = s->stat.sz_tried;
> > + nr_lru_sorted_hot_regions = s->stat.nr_applied;
> > + bytes_lru_sorted_hot_regions = s->stat.sz_applied;
> > + nr_hot_quota_exceeds = s->stat.qt_exceeds;
> > + } else if (s->action == DAMOS_LRU_DEPRIO) {
> > + nr_lru_sort_tried_cold_regions = s->stat.nr_tried;
> > + bytes_lru_sort_tried_cold_regions = s->stat.sz_tried;
> > + nr_lru_sorted_cold_regions = s->stat.nr_applied;
> > + bytes_lru_sorted_cold_regions = s->stat.sz_applied;
> > + nr_cold_quota_exceeds = s->stat.qt_exceeds;
> > + }
> > + }
> > +
> > + return damon_lru_sort_handle_commit_inputs();
> > +}
> > +
> > +static int damon_lru_sort_after_wmarks_check(struct damon_ctx *c)
> > +{
> > + return damon_lru_sort_handle_commit_inputs();
> > +}
> > +
> > +static int __init damon_lru_sort_init(void)
> > +{
> > + ctx = damon_new_ctx();
> > + if (!ctx)
> > + return -ENOMEM;
> > +
> > + if (damon_select_ops(ctx, DAMON_OPS_PADDR))
> > + return -EINVAL;
> > +
> > + ctx->callback.after_wmarks_check = damon_lru_sort_after_wmarks_check;
> > + ctx->callback.after_aggregation = damon_lru_sort_after_aggregation;
> > +
> > + target = damon_new_target();
> > + if (!target) {
> > + damon_destroy_ctx(ctx);
> > + return -ENOMEM;
> > + }
> > + damon_add_target(ctx, target);
> > +
> > + schedule_delayed_work(&damon_lru_sort_timer, 0);
> > +
> > + damon_lru_sort_initialized = true;
> > + return 0;
> > +}
> > +
> > +module_init(damon_lru_sort_init);
> > --
> > 2.25.1
> >
Thanks
Barry
Hi Barry,
On Thu, 1 Sep 2022 14:21:21 +1200 Barry Song <[email protected]> wrote:
> On Thu, Sep 1, 2022 at 2:03 PM Barry Song <[email protected]> wrote:
> >
> > On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
> > >
> > > Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> > > 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> > > the hotness/coldness thresholds, CPU quota, and watermarks could be
> > > challenging for some users. To make the scheme easy to be used without
> > > complex tuning for common situations, this commit implements a static
> > > kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> > > 'LRU_DEPRIO' DAMOS actions.
> > >
> > > It proactively sorts LRU-lists using DAMON with conservatively chosen
> > > default values of the parameters. That is, the module under its default
> > > parameters will make no harm for common situations but provide some
> > > level of efficiency improvements for systems having clear hot/cold
> > > access pattern under a level of memory pressure while consuming only a
> > > limited small portion of CPU time.
> >
>
> Hi SeongJae,
> While I believe DAMON pro-active reclamation and LRU-SORT can benefit the system
> by either swapping out cold pages earlier and sorting LRU lists before
> system has high
> memory pressure, I am still not convinced the improvement really comes from the
> identification of cold and hot pages by DAMON.
>
> My guess is that even if we randomly pick some regions in memory and do the same
> thing in the kernel, we can also see the improvement.
>
> As we actually depend on two facts to benefit from DAMON
> 1. locality
> while virtual address might have some locality, physical address seems
> not. for example,
> address A might be mapped by facebook, address A + 4096 could be
> mapped by youtube.
> There is nothing which can stop contiguous physical addresses from
> being mapped by
> completely irrelevant applications. so regions based on paddr seems pointless.
>
> 2. accuration
> As I have reported it is very hard for damon to accurately track
> virtual address since
> virtual space is so huge:
> https://lore.kernel.org/lkml/CAGsJ_4x_k9009HwpTswEq1ut_co8XYdpZ9k0BVW=0=HRiifxkA@mail.gmail.com/
> I believe it is also true for paddr since paddr has much worse
> locality than vaddr.
> so we probably need a lot of regions, ideally, one region for each page.
>
> To me, it seems neither of these two facts are true. So I am more
> willing to believe
> that the benefits come from areas picked randomly.
>
> Am I missing something?
Thank you for the questions.
As you mentioned, DAMON assumes spatial and temporal locality, to trade
accuracy for lower overhead[1]. That is, DAMON believes some memory regions
would have pages that accessed in similar frequency for similar time duration.
Therefore if the access pattern of the system is really chaotic, that is, if
every adjacent page have very different access frequency or the access
frequency changes very frequently, DAMON's accuracy would be bad. But, would
such access pattern really common in the real world? Given the Pareto
principle[2], I think that's not always true. After all, many of kernel
mechanisms including the pseudo-LRU-based reclamation and the readahead assumes
some locality and makes good effect.
If your system has too low locality and therefore DAMON doesn't provide good
enough accuracy, you could increase the accuracy by setting the upperbound of
the monitoring overhead higher. For DAMOS schemes like DAMON_RECLAIM or
DAMON_LRU_SORT, you could also increase the minimum age of the target access
pattern. If the access pattern is really chaotic, DAMON wouldn't show the
regions having the specific access pattern for long time. Actually, definition
of the age and use of it means you believe the system's access pattern is not
that chaotic but has at least temporal locality.
It's true that DAMON doesn't monitor access pattern in page granularity, and
therefore it could report some cold pages as hot, and vice versa. However, I'd
say the benefit of making right decision for huge number of pages outweighs the
risk of making wrong decision for few pages in many cases.
After all, it shows some benefit on my test environments and some production
systems. I haven't compared that against random pageout or random lru sorting,
though.
Nevertheless, DAMON has so many rooms for improvement, including the accuracy.
I want to improve the accuracy while keeping the overhead low. Also, I know
that there are people who willing to do page-granularity monitoring though it
could incur high monitoring overhead. As a part of the DAMON accuracy
improvement plan, to use that as a comparison target, and to convince such
people, I added the page granularity monitoring feature of DAMON to my todo
list. I haven't had a time for prioritizing that yet, though, as I haven't
heard some clear voice of users for that. I hope the DAMON Beer/Coffee/Tea
Chat Series to be a place to hear such voices.
[1] https://docs.kernel.org/mm/damon/design.html#address-space-independent-core-mechanisms
[2] https://en.wikipedia.org/wiki/Pareto_principle
Thanks,
SJ
On Fri, Sep 2, 2022 at 5:11 AM SeongJae Park <[email protected]> wrote:
>
> Hi Barry,
>
>
> On Thu, 1 Sep 2022 14:21:21 +1200 Barry Song <[email protected]> wrote:
>
> > On Thu, Sep 1, 2022 at 2:03 PM Barry Song <[email protected]> wrote:
> > >
> > > On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
> > > >
> > > > Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> > > > 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> > > > the hotness/coldness thresholds, CPU quota, and watermarks could be
> > > > challenging for some users. To make the scheme easy to be used without
> > > > complex tuning for common situations, this commit implements a static
> > > > kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> > > > 'LRU_DEPRIO' DAMOS actions.
> > > >
> > > > It proactively sorts LRU-lists using DAMON with conservatively chosen
> > > > default values of the parameters. That is, the module under its default
> > > > parameters will make no harm for common situations but provide some
> > > > level of efficiency improvements for systems having clear hot/cold
> > > > access pattern under a level of memory pressure while consuming only a
> > > > limited small portion of CPU time.
> > >
> >
> > Hi SeongJae,
> > While I believe DAMON pro-active reclamation and LRU-SORT can benefit the system
> > by either swapping out cold pages earlier and sorting LRU lists before
> > system has high
> > memory pressure, I am still not convinced the improvement really comes from the
> > identification of cold and hot pages by DAMON.
> >
> > My guess is that even if we randomly pick some regions in memory and do the same
> > thing in the kernel, we can also see the improvement.
> >
> > As we actually depend on two facts to benefit from DAMON
> > 1. locality
> > while virtual address might have some locality, physical address seems
> > not. for example,
> > address A might be mapped by facebook, address A + 4096 could be
> > mapped by youtube.
> > There is nothing which can stop contiguous physical addresses from
> > being mapped by
> > completely irrelevant applications. so regions based on paddr seems pointless.
> >
> > 2. accuration
> > As I have reported it is very hard for damon to accurately track
> > virtual address since
> > virtual space is so huge:
> > https://lore.kernel.org/lkml/CAGsJ_4x_k9009HwpTswEq1ut_co8XYdpZ9k0BVW=0=HRiifxkA@mail.gmail.com/
> > I believe it is also true for paddr since paddr has much worse
> > locality than vaddr.
> > so we probably need a lot of regions, ideally, one region for each page.
> >
> > To me, it seems neither of these two facts are true. So I am more
> > willing to believe
> > that the benefits come from areas picked randomly.
> >
> > Am I missing something?
>
> Thank you for the questions.
>
> As you mentioned, DAMON assumes spatial and temporal locality, to trade
> accuracy for lower overhead[1]. That is, DAMON believes some memory regions
> would have pages that accessed in similar frequency for similar time duration.
> Therefore if the access pattern of the system is really chaotic, that is, if
> every adjacent page have very different access frequency or the access
> frequency changes very frequently, DAMON's accuracy would be bad. But, would
> such access pattern really common in the real world? Given the Pareto
> principle[2], I think that's not always true. After all, many of kernel
> mechanisms including the pseudo-LRU-based reclamation and the readahead assumes
> some locality and makes good effect.
+ yu zhao
I do believe we have some locality in virtual addresses as they are in
the same application.
that is why we can "exploit locality in rmap" here:
https://lore.kernel.org/linux-mm/[email protected]/
But for paddr, i doubt it is true as processes use page faults to get
pages from buddy
mainly in low order like zero.
>
> If your system has too low locality and therefore DAMON doesn't provide good
> enough accuracy, you could increase the accuracy by setting the upperbound of
> the monitoring overhead higher. For DAMOS schemes like DAMON_RECLAIM or
> DAMON_LRU_SORT, you could also increase the minimum age of the target access
> pattern. If the access pattern is really chaotic, DAMON wouldn't show the
> regions having the specific access pattern for long time. Actually, definition
> of the age and use of it means you believe the system's access pattern is not
> that chaotic but has at least temporal locality.
>
> It's true that DAMON doesn't monitor access pattern in page granularity, and
> therefore it could report some cold pages as hot, and vice versa. However, I'd
> say the benefit of making right decision for huge number of pages outweighs the
> risk of making wrong decision for few pages in many cases.
>
> After all, it shows some benefit on my test environments and some production
> systems. I haven't compared that against random pageout or random lru sorting,
> though.
>
> Nevertheless, DAMON has so many rooms for improvement, including the accuracy.
> I want to improve the accuracy while keeping the overhead low. Also, I know
> that there are people who willing to do page-granularity monitoring though it
> could incur high monitoring overhead. As a part of the DAMON accuracy
> improvement plan, to use that as a comparison target, and to convince such
> people, I added the page granularity monitoring feature of DAMON to my todo
> list. I haven't had a time for prioritizing that yet, though, as I haven't
> heard some clear voice of users for that. I hope the DAMON Beer/Coffee/Tea
> Chat Series to be a place to hear such voices.
is it possible for us to leverage the idea from "mm: multi-gen LRU:
support page table walks"
https://lore.kernel.org/linux-mm/[email protected]/
we pro-actively scan the virtual address space of those processes
which have been really
executed then get LRU sorted earlier?
>
> [1] https://docs.kernel.org/mm/damon/design.html#address-space-independent-core-mechanisms
> [2] https://en.wikipedia.org/wiki/Pareto_principle
>
>
> Thanks,
> SJ
Thanks
Barry
On Fri, 2 Sep 2022 09:40:10 +1200 Barry Song <[email protected]> wrote:
> On Fri, Sep 2, 2022 at 5:11 AM SeongJae Park <[email protected]> wrote:
> >
> > Hi Barry,
> >
> >
> > On Thu, 1 Sep 2022 14:21:21 +1200 Barry Song <[email protected]> wrote:
> >
> > > On Thu, Sep 1, 2022 at 2:03 PM Barry Song <[email protected]> wrote:
> > > >
> > > > On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
> > > > >
> > > > > Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> > > > > 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> > > > > the hotness/coldness thresholds, CPU quota, and watermarks could be
> > > > > challenging for some users. To make the scheme easy to be used without
> > > > > complex tuning for common situations, this commit implements a static
> > > > > kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> > > > > 'LRU_DEPRIO' DAMOS actions.
> > > > >
> > > > > It proactively sorts LRU-lists using DAMON with conservatively chosen
> > > > > default values of the parameters. That is, the module under its default
> > > > > parameters will make no harm for common situations but provide some
> > > > > level of efficiency improvements for systems having clear hot/cold
> > > > > access pattern under a level of memory pressure while consuming only a
> > > > > limited small portion of CPU time.
> > > >
> > >
> > > Hi SeongJae,
> > > While I believe DAMON pro-active reclamation and LRU-SORT can benefit the system
> > > by either swapping out cold pages earlier and sorting LRU lists before
> > > system has high
> > > memory pressure, I am still not convinced the improvement really comes from the
> > > identification of cold and hot pages by DAMON.
> > >
> > > My guess is that even if we randomly pick some regions in memory and do the same
> > > thing in the kernel, we can also see the improvement.
> > >
> > > As we actually depend on two facts to benefit from DAMON
> > > 1. locality
> > > while virtual address might have some locality, physical address seems
> > > not. for example,
> > > address A might be mapped by facebook, address A + 4096 could be
> > > mapped by youtube.
> > > There is nothing which can stop contiguous physical addresses from
> > > being mapped by
> > > completely irrelevant applications. so regions based on paddr seems pointless.
> > >
> > > 2. accuration
> > > As I have reported it is very hard for damon to accurately track
> > > virtual address since
> > > virtual space is so huge:
> > > https://lore.kernel.org/lkml/CAGsJ_4x_k9009HwpTswEq1ut_co8XYdpZ9k0BVW=0=HRiifxkA@mail.gmail.com/
> > > I believe it is also true for paddr since paddr has much worse
> > > locality than vaddr.
> > > so we probably need a lot of regions, ideally, one region for each page.
> > >
> > > To me, it seems neither of these two facts are true. So I am more
> > > willing to believe
> > > that the benefits come from areas picked randomly.
> > >
> > > Am I missing something?
> >
> > Thank you for the questions.
> >
> > As you mentioned, DAMON assumes spatial and temporal locality, to trade
> > accuracy for lower overhead[1]. That is, DAMON believes some memory regions
> > would have pages that accessed in similar frequency for similar time duration.
> > Therefore if the access pattern of the system is really chaotic, that is, if
> > every adjacent page have very different access frequency or the access
> > frequency changes very frequently, DAMON's accuracy would be bad. But, would
> > such access pattern really common in the real world? Given the Pareto
> > principle[2], I think that's not always true. After all, many of kernel
> > mechanisms including the pseudo-LRU-based reclamation and the readahead assumes
> > some locality and makes good effect.
>
> + yu zhao
>
> I do believe we have some locality in virtual addresses as they are in
> the same application.
> that is why we can "exploit locality in rmap" here:
> https://lore.kernel.org/linux-mm/[email protected]/
>
> But for paddr, i doubt it is true as processes use page faults to get
> pages from buddy
> mainly in low order like zero.
Well, what I can tell for now is that it would depend on the specific system
and workload, but I found some production systems that have such kind of
physical address space locality.
>
> >
> > If your system has too low locality and therefore DAMON doesn't provide good
> > enough accuracy, you could increase the accuracy by setting the upperbound of
> > the monitoring overhead higher. For DAMOS schemes like DAMON_RECLAIM or
> > DAMON_LRU_SORT, you could also increase the minimum age of the target access
> > pattern. If the access pattern is really chaotic, DAMON wouldn't show the
> > regions having the specific access pattern for long time. Actually, definition
> > of the age and use of it means you believe the system's access pattern is not
> > that chaotic but has at least temporal locality.
> >
> > It's true that DAMON doesn't monitor access pattern in page granularity, and
> > therefore it could report some cold pages as hot, and vice versa. However, I'd
> > say the benefit of making right decision for huge number of pages outweighs the
> > risk of making wrong decision for few pages in many cases.
> >
> > After all, it shows some benefit on my test environments and some production
> > systems. I haven't compared that against random pageout or random lru sorting,
> > though.
> >
> > Nevertheless, DAMON has so many rooms for improvement, including the accuracy.
> > I want to improve the accuracy while keeping the overhead low. Also, I know
> > that there are people who willing to do page-granularity monitoring though it
> > could incur high monitoring overhead. As a part of the DAMON accuracy
> > improvement plan, to use that as a comparison target, and to convince such
> > people, I added the page granularity monitoring feature of DAMON to my todo
> > list. I haven't had a time for prioritizing that yet, though, as I haven't
> > heard some clear voice of users for that. I hope the DAMON Beer/Coffee/Tea
> > Chat Series to be a place to hear such voices.
>
> is it possible for us to leverage the idea from "mm: multi-gen LRU:
> support page table walks"
>
> https://lore.kernel.org/linux-mm/[email protected]/
>
> we pro-actively scan the virtual address space of those processes
> which have been really
> executed then get LRU sorted earlier?
I didn't read MGLRU patchset thoroughly, but, maybe?
Thanks,
SJ
>
> >
> > [1] https://docs.kernel.org/mm/damon/design.html#address-space-independent-core-mechanisms
> > [2] https://en.wikipedia.org/wiki/Pareto_principle
> >
> >
> > Thanks,
> > SJ
>
> Thanks
> Barry
On Fri, Sep 2, 2022 at 10:11 AM SeongJae Park <[email protected]> wrote:
>
> On Fri, 2 Sep 2022 09:40:10 +1200 Barry Song <[email protected]> wrote:
>
> > On Fri, Sep 2, 2022 at 5:11 AM SeongJae Park <[email protected]> wrote:
> > >
> > > Hi Barry,
> > >
> > >
> > > On Thu, 1 Sep 2022 14:21:21 +1200 Barry Song <[email protected]> wrote:
> > >
> > > > On Thu, Sep 1, 2022 at 2:03 PM Barry Song <[email protected]> wrote:
> > > > >
> > > > > On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
> > > > > >
> > > > > > Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> > > > > > 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> > > > > > the hotness/coldness thresholds, CPU quota, and watermarks could be
> > > > > > challenging for some users. To make the scheme easy to be used without
> > > > > > complex tuning for common situations, this commit implements a static
> > > > > > kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> > > > > > 'LRU_DEPRIO' DAMOS actions.
> > > > > >
> > > > > > It proactively sorts LRU-lists using DAMON with conservatively chosen
> > > > > > default values of the parameters. That is, the module under its default
> > > > > > parameters will make no harm for common situations but provide some
> > > > > > level of efficiency improvements for systems having clear hot/cold
> > > > > > access pattern under a level of memory pressure while consuming only a
> > > > > > limited small portion of CPU time.
> > > > >
> > > >
> > > > Hi SeongJae,
> > > > While I believe DAMON pro-active reclamation and LRU-SORT can benefit the system
> > > > by either swapping out cold pages earlier and sorting LRU lists before
> > > > system has high
> > > > memory pressure, I am still not convinced the improvement really comes from the
> > > > identification of cold and hot pages by DAMON.
> > > >
> > > > My guess is that even if we randomly pick some regions in memory and do the same
> > > > thing in the kernel, we can also see the improvement.
> > > >
> > > > As we actually depend on two facts to benefit from DAMON
> > > > 1. locality
> > > > while virtual address might have some locality, physical address seems
> > > > not. for example,
> > > > address A might be mapped by facebook, address A + 4096 could be
> > > > mapped by youtube.
> > > > There is nothing which can stop contiguous physical addresses from
> > > > being mapped by
> > > > completely irrelevant applications. so regions based on paddr seems pointless.
> > > >
> > > > 2. accuration
> > > > As I have reported it is very hard for damon to accurately track
> > > > virtual address since
> > > > virtual space is so huge:
> > > > https://lore.kernel.org/lkml/CAGsJ_4x_k9009HwpTswEq1ut_co8XYdpZ9k0BVW=0=HRiifxkA@mail.gmail.com/
> > > > I believe it is also true for paddr since paddr has much worse
> > > > locality than vaddr.
> > > > so we probably need a lot of regions, ideally, one region for each page.
> > > >
> > > > To me, it seems neither of these two facts are true. So I am more
> > > > willing to believe
> > > > that the benefits come from areas picked randomly.
> > > >
> > > > Am I missing something?
> > >
> > > Thank you for the questions.
> > >
> > > As you mentioned, DAMON assumes spatial and temporal locality, to trade
> > > accuracy for lower overhead[1]. That is, DAMON believes some memory regions
> > > would have pages that accessed in similar frequency for similar time duration.
> > > Therefore if the access pattern of the system is really chaotic, that is, if
> > > every adjacent page have very different access frequency or the access
> > > frequency changes very frequently, DAMON's accuracy would be bad. But, would
> > > such access pattern really common in the real world? Given the Pareto
> > > principle[2], I think that's not always true. After all, many of kernel
> > > mechanisms including the pseudo-LRU-based reclamation and the readahead assumes
> > > some locality and makes good effect.
> >
> > + yu zhao
> >
> > I do believe we have some locality in virtual addresses as they are in
> > the same application.
> > that is why we can "exploit locality in rmap" here:
> > https://lore.kernel.org/linux-mm/[email protected]/
> >
> > But for paddr, i doubt it is true as processes use page faults to get
> > pages from buddy
> > mainly in low order like zero.
>
> Well, what I can tell for now is that it would depend on the specific system
> and workload, but I found some production systems that have such kind of
> physical address space locality.
yep. I guess for mmu-less systems, spatial locality is more likely to be true.
for mmu system, workload using 2MB or 1GB THP will show some locality in
physical address as well. but THP_SWP will swap them as a whole, so it
seems locality inside a THP doesn't help? as we are not splitting sub pages
of THP for reclamation?
>
> >
> > >
> > > If your system has too low locality and therefore DAMON doesn't provide good
> > > enough accuracy, you could increase the accuracy by setting the upperbound of
> > > the monitoring overhead higher. For DAMOS schemes like DAMON_RECLAIM or
> > > DAMON_LRU_SORT, you could also increase the minimum age of the target access
> > > pattern. If the access pattern is really chaotic, DAMON wouldn't show the
> > > regions having the specific access pattern for long time. Actually, definition
> > > of the age and use of it means you believe the system's access pattern is not
> > > that chaotic but has at least temporal locality.
> > >
> > > It's true that DAMON doesn't monitor access pattern in page granularity, and
> > > therefore it could report some cold pages as hot, and vice versa. However, I'd
> > > say the benefit of making right decision for huge number of pages outweighs the
> > > risk of making wrong decision for few pages in many cases.
> > >
> > > After all, it shows some benefit on my test environments and some production
> > > systems. I haven't compared that against random pageout or random lru sorting,
> > > though.
> > >
> > > Nevertheless, DAMON has so many rooms for improvement, including the accuracy.
> > > I want to improve the accuracy while keeping the overhead low. Also, I know
> > > that there are people who willing to do page-granularity monitoring though it
> > > could incur high monitoring overhead. As a part of the DAMON accuracy
> > > improvement plan, to use that as a comparison target, and to convince such
> > > people, I added the page granularity monitoring feature of DAMON to my todo
> > > list. I haven't had a time for prioritizing that yet, though, as I haven't
> > > heard some clear voice of users for that. I hope the DAMON Beer/Coffee/Tea
> > > Chat Series to be a place to hear such voices.
> >
> > is it possible for us to leverage the idea from "mm: multi-gen LRU:
> > support page table walks"
> >
> > https://lore.kernel.org/linux-mm/[email protected]/
> >
> > we pro-actively scan the virtual address space of those processes
> > which have been really
> > executed then get LRU sorted earlier?
>
> I didn't read MGLRU patchset thoroughly, but, maybe?
>
Yes, please read MGLRU, SeongJae. I find some ideas in MGLRU are
really good. we might
leverage those to improve DAMON as well :-)
>
> Thanks,
> SJ
>
> >
> > >
> > > [1] https://docs.kernel.org/mm/damon/design.html#address-space-independent-core-mechanisms
> > > [2] https://en.wikipedia.org/wiki/Pareto_principle
> > >
> > >
> > > Thanks,
> > > SJ
> >
Thanks
Barry
On Fri, Sep 2, 2022 at 10:11 AM SeongJae Park <[email protected]> wrote:
>
> On Fri, 2 Sep 2022 09:40:10 +1200 Barry Song <[email protected]> wrote:
>
> > On Fri, Sep 2, 2022 at 5:11 AM SeongJae Park <[email protected]> wrote:
> > >
> > > Hi Barry,
> > >
> > >
> > > On Thu, 1 Sep 2022 14:21:21 +1200 Barry Song <[email protected]> wrote:
> > >
> > > > On Thu, Sep 1, 2022 at 2:03 PM Barry Song <[email protected]> wrote:
> > > > >
> > > > > On Tue, Jun 14, 2022 at 10:01 AM SeongJae Park <[email protected]> wrote:
> > > > > >
> > > > > > Users can do data access-aware LRU-lists sorting using 'LRU_PRIO' and
> > > > > > 'LRU_DEPRIO' DAMOS actions. However, finding best parameters including
> > > > > > the hotness/coldness thresholds, CPU quota, and watermarks could be
> > > > > > challenging for some users. To make the scheme easy to be used without
> > > > > > complex tuning for common situations, this commit implements a static
> > > > > > kernel module called 'DAMON_LRU_SORT' using the 'LRU_PRIO' and
> > > > > > 'LRU_DEPRIO' DAMOS actions.
> > > > > >
> > > > > > It proactively sorts LRU-lists using DAMON with conservatively chosen
> > > > > > default values of the parameters. That is, the module under its default
> > > > > > parameters will make no harm for common situations but provide some
> > > > > > level of efficiency improvements for systems having clear hot/cold
> > > > > > access pattern under a level of memory pressure while consuming only a
> > > > > > limited small portion of CPU time.
> > > > >
> > > >
> > > > Hi SeongJae,
> > > > While I believe DAMON pro-active reclamation and LRU-SORT can benefit the system
> > > > by either swapping out cold pages earlier and sorting LRU lists before
> > > > system has high
> > > > memory pressure, I am still not convinced the improvement really comes from the
> > > > identification of cold and hot pages by DAMON.
> > > >
> > > > My guess is that even if we randomly pick some regions in memory and do the same
> > > > thing in the kernel, we can also see the improvement.
> > > >
> > > > As we actually depend on two facts to benefit from DAMON
> > > > 1. locality
> > > > while virtual address might have some locality, physical address seems
> > > > not. for example,
> > > > address A might be mapped by facebook, address A + 4096 could be
> > > > mapped by youtube.
> > > > There is nothing which can stop contiguous physical addresses from
> > > > being mapped by
> > > > completely irrelevant applications. so regions based on paddr seems pointless.
> > > >
> > > > 2. accuration
> > > > As I have reported it is very hard for damon to accurately track
> > > > virtual address since
> > > > virtual space is so huge:
> > > > https://lore.kernel.org/lkml/CAGsJ_4x_k9009HwpTswEq1ut_co8XYdpZ9k0BVW=0=HRiifxkA@mail.gmail.com/
> > > > I believe it is also true for paddr since paddr has much worse
> > > > locality than vaddr.
> > > > so we probably need a lot of regions, ideally, one region for each page.
> > > >
> > > > To me, it seems neither of these two facts are true. So I am more
> > > > willing to believe
> > > > that the benefits come from areas picked randomly.
> > > >
> > > > Am I missing something?
> > >
> > > Thank you for the questions.
> > >
> > > As you mentioned, DAMON assumes spatial and temporal locality, to trade
> > > accuracy for lower overhead[1]. That is, DAMON believes some memory regions
> > > would have pages that accessed in similar frequency for similar time duration.
> > > Therefore if the access pattern of the system is really chaotic, that is, if
> > > every adjacent page have very different access frequency or the access
> > > frequency changes very frequently, DAMON's accuracy would be bad. But, would
> > > such access pattern really common in the real world? Given the Pareto
> > > principle[2], I think that's not always true. After all, many of kernel
> > > mechanisms including the pseudo-LRU-based reclamation and the readahead assumes
> > > some locality and makes good effect.
> >
> > + yu zhao
> >
> > I do believe we have some locality in virtual addresses as they are in
> > the same application.
> > that is why we can "exploit locality in rmap" here:
> > https://lore.kernel.org/linux-mm/[email protected]/
> >
> > But for paddr, i doubt it is true as processes use page faults to get
> > pages from buddy
> > mainly in low order like zero.
>
> Well, what I can tell for now is that it would depend on the specific system
> and workload, but I found some production systems that have such kind of
> physical address space locality.
>
yep. I guess for mmu-less systems, spatial locality is more likely to be true.
for mmu system, workload using 2MB or 1GB THP won't show locality in
physical address as well. but THP_SWP will swap them as a whole, so it
seems locality inside a THP
> >
> > >
> > > If your system has too low locality and therefore DAMON doesn't provide good
> > > enough accuracy, you could increase the accuracy by setting the upperbound of
> > > the monitoring overhead higher. For DAMOS schemes like DAMON_RECLAIM or
> > > DAMON_LRU_SORT, you could also increase the minimum age of the target access
> > > pattern. If the access pattern is really chaotic, DAMON wouldn't show the
> > > regions having the specific access pattern for long time. Actually, definition
> > > of the age and use of it means you believe the system's access pattern is not
> > > that chaotic but has at least temporal locality.
> > >
> > > It's true that DAMON doesn't monitor access pattern in page granularity, and
> > > therefore it could report some cold pages as hot, and vice versa. However, I'd
> > > say the benefit of making right decision for huge number of pages outweighs the
> > > risk of making wrong decision for few pages in many cases.
> > >
> > > After all, it shows some benefit on my test environments and some production
> > > systems. I haven't compared that against random pageout or random lru sorting,
> > > though.
> > >
> > > Nevertheless, DAMON has so many rooms for improvement, including the accuracy.
> > > I want to improve the accuracy while keeping the overhead low. Also, I know
> > > that there are people who willing to do page-granularity monitoring though it
> > > could incur high monitoring overhead. As a part of the DAMON accuracy
> > > improvement plan, to use that as a comparison target, and to convince such
> > > people, I added the page granularity monitoring feature of DAMON to my todo
> > > list. I haven't had a time for prioritizing that yet, though, as I haven't
> > > heard some clear voice of users for that. I hope the DAMON Beer/Coffee/Tea
> > > Chat Series to be a place to hear such voices.
> >
> > is it possible for us to leverage the idea from "mm: multi-gen LRU:
> > support page table walks"
> >
> > https://lore.kernel.org/linux-mm/[email protected]/
> >
> > we pro-actively scan the virtual address space of those processes
> > which have been really
> > executed then get LRU sorted earlier?
>
> I didn't read MGLRU patchset thoroughly, but, maybe?
>
>
> Thanks,
> SJ
>
> >
> > >
> > > [1] https://docs.kernel.org/mm/damon/design.html#address-space-independent-core-mechanisms
> > > [2] https://en.wikipedia.org/wiki/Pareto_principle
> > >
> > >
> > > Thanks,
> > > SJ
> >
> > Thanks
> > Barry