Changes from RFC
(https://lore.kernel.org/damon/[email protected]/)
- Rebase on latest mm-unstable
- Minor wordsmithing of coverletter
DAMON checks the access to each region for every sampling interval, increase
the access rate counter of the region, namely nr_accesses, if the access was
made. For every aggregation interval, the counter is reset. The counter is
exposed to users to be used as a metric showing the relative access rate
(frequency) of each region. In other words, DAMON provides access rate of each
region in every aggregation interval. The aggregation avoids temporal access
pattern changes making things confusing. However, this also makes a few
DAMON-related operations to unnecessarily need to be aligned to the aggregation
interval. This can restrict the flexibility of DAMON applications, especially
when the aggregation interval is huge.
To provide the monitoring results in finer-grained timing while keeping
handling of temporal access pattern change, this patchset implements a
pseudo-moving sum based access rate metric. It is pseudo-moving sum because
strict moving sum implementation would need to keep all values for last time
window, and that could incur high overhead of there could be arbitrary number
of values in a time window. Especially in case of the nr_accesses, since the
sampling interval and aggregation interval can arbitrarily set and the past
values should be maintained for every region, it could be risky. The
pseudo-moving sum assumes there were no temporal access pattern change in last
discrete time window to remove the needs for keeping the list of the last time
window values. As a result, it beocmes not strict moving sum implementation,
but provides a reasonable accuracy.
Also, it keeps an important property of the moving sum. That is, the moving
sum becomes same to discrete-window based sum at the time that aligns to the
time window. This means using the pseudo moving sum based nr_accesses makes no
change to users who shows the value for every aggregation interval.
Patches Sequence
----------------
The sequence of the patches is as follows. The first four patches are
for preparation of the change. The first two (patches 1 and 2)
implements a helper function for nr_accesses update and eliminate corner
case that skips use of the function, respectively. Following two
(patches 3 and 4) respectively implement the pseudo-moving sum function
and its simple unit test case.
Two patches for making DAMON to use the pseudo-moving sum follow. The
fifthe one (patch 5) introduces a new field for representing the
pseudo-moving sum-based access rate of each region, and the sixth one
makes the new representation to actually updated with the pseudo-moving
sum function.
Last two patches (patches 7 and 8) makes followup fixes for skipping
unnecessary updates and marking the moving sum function as static,
respectively.
SeongJae Park (8):
mm/damon/core: define and use a dedicated function for region access
rate update
mm/damon/vaddr: call damon_update_region_access_rate() always
mm/damon/core: implement a pseudo-moving sum function
mm/damon/core-test: add a unit test for damon_moving_sum()
mm/damon/core: introduce nr_accesses_bp
mm/damon/core: use pseudo-moving sum for nr_accesses_bp
mm/damon/core: skip updating nr_accesses_bp for each aggregation
interval
mm/damon/core: mark damon_moving_sum() as a static function
include/linux/damon.h | 16 +++++++++-
mm/damon/core-test.h | 21 ++++++++++++
mm/damon/core.c | 74 +++++++++++++++++++++++++++++++++++++++++++
mm/damon/paddr.c | 11 +++----
mm/damon/vaddr.c | 22 +++++++------
5 files changed, 128 insertions(+), 16 deletions(-)
base-commit: a5b7405a0eaa74d23547ede9c3820f01ee0a2c13
--
2.25.1
The function is used by only mm/damon/core.c. Mark it as a static
function.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/damon.h | 2 --
mm/damon/core.c | 2 +-
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 0fe13482df63..491fdd3e4c76 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -632,8 +632,6 @@ void damon_add_region(struct damon_region *r, struct damon_target *t);
void damon_destroy_region(struct damon_region *r, struct damon_target *t);
int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
unsigned int nr_ranges);
-unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
- unsigned int len_window, unsigned int new_value);
void damon_update_region_access_rate(struct damon_region *r, bool accessed,
struct damon_attrs *attrs);
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 45cc108c0fe1..b15cf47d2d29 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -1587,7 +1587,7 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t,
*
* Return: Pseudo-moving average after getting the @new_value.
*/
-unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
+static unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
unsigned int len_window, unsigned int new_value)
{
return mvsum - nomvsum / len_window + new_value;
--
2.25.1
Add yet another representation of the access rate of each region, namely
nr_accesses_bp. It is just same to the nr_accesses but represents the
value in basis point (1 in 10,000), and updated at once in every
aggregation interval. That is, moving_accesses_bp is just nr_accesses *
10000. This may seems useless at the moment. However, it will be
useful for representing less than one nr_accesses value that will be
needed to make moving sum-based nr_accesses.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/damon.h | 5 +++++
mm/damon/core-test.h | 5 +++++
mm/damon/core.c | 6 ++++++
3 files changed, 16 insertions(+)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 487a545a11b4..15f24b23c9a0 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -40,6 +40,7 @@ struct damon_addr_range {
* @ar: The address range of the region.
* @sampling_addr: Address of the sample for the next access check.
* @nr_accesses: Access frequency of this region.
+ * @nr_accesses_bp: @nr_accesses in basis point (0.01%).
* @list: List head for siblings.
* @age: Age of this region.
*
@@ -49,6 +50,9 @@ struct damon_addr_range {
* not be done with direct access but with the helper function,
* damon_update_region_access_rate().
*
+ * @nr_accesses_bp is another representation of @nr_accesses in basis point
+ * (1 in 10,000) that updated every aggregation interval.
+ *
* @age is initially zero, increased for each aggregation interval, and reset
* to zero again if the access frequency is significantly changed. If two
* regions are merged into a new region, both @nr_accesses and @age of the new
@@ -58,6 +62,7 @@ struct damon_region {
struct damon_addr_range ar;
unsigned long sampling_addr;
unsigned int nr_accesses;
+ unsigned int nr_accesses_bp;
struct list_head list;
unsigned int age;
diff --git a/mm/damon/core-test.h b/mm/damon/core-test.h
index c539f0e8377e..79f1f12e0dd5 100644
--- a/mm/damon/core-test.h
+++ b/mm/damon/core-test.h
@@ -94,6 +94,7 @@ static void damon_test_aggregate(struct kunit *test)
for (ir = 0; ir < 3; ir++) {
r = damon_new_region(saddr[it][ir], eaddr[it][ir]);
r->nr_accesses = accesses[it][ir];
+ r->nr_accesses_bp = accesses[it][ir] * 10000;
damon_add_region(r, t);
}
it++;
@@ -147,9 +148,11 @@ static void damon_test_merge_two(struct kunit *test)
t = damon_new_target();
r = damon_new_region(0, 100);
r->nr_accesses = 10;
+ r->nr_accesses_bp = 100000;
damon_add_region(r, t);
r2 = damon_new_region(100, 300);
r2->nr_accesses = 20;
+ r2->nr_accesses_bp = 200000;
damon_add_region(r2, t);
damon_merge_two_regions(t, r, r2);
@@ -196,6 +199,7 @@ static void damon_test_merge_regions_of(struct kunit *test)
for (i = 0; i < ARRAY_SIZE(sa); i++) {
r = damon_new_region(sa[i], ea[i]);
r->nr_accesses = nrs[i];
+ r->nr_accesses_bp = nrs[i] * 10000;
damon_add_region(r, t);
}
@@ -297,6 +301,7 @@ static void damon_test_update_monitoring_result(struct kunit *test)
struct damon_region *r = damon_new_region(3, 7);
r->nr_accesses = 15;
+ r->nr_accesses_bp = 150000;
r->age = 20;
new_attrs = (struct damon_attrs){
diff --git a/mm/damon/core.c b/mm/damon/core.c
index b005dc15009f..ce85c00b0a4c 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -128,6 +128,7 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end)
region->ar.start = start;
region->ar.end = end;
region->nr_accesses = 0;
+ region->nr_accesses_bp = 0;
INIT_LIST_HEAD(®ion->list);
region->age = 0;
@@ -508,6 +509,7 @@ static void damon_update_monitoring_result(struct damon_region *r,
{
r->nr_accesses = damon_nr_accesses_for_new_attrs(r->nr_accesses,
old_attrs, new_attrs);
+ r->nr_accesses_bp = r->nr_accesses * 10000;
r->age = damon_age_for_new_attrs(r->age, old_attrs, new_attrs);
}
@@ -1115,6 +1117,7 @@ static void damon_merge_two_regions(struct damon_target *t,
l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
(sz_l + sz_r);
+ l->nr_accesses_bp = l->nr_accesses * 10000;
l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);
l->ar.end = r->ar.end;
damon_destroy_region(r, t);
@@ -1138,6 +1141,8 @@ static void damon_merge_regions_of(struct damon_target *t, unsigned int thres,
else
r->age++;
+ r->nr_accesses_bp = r->nr_accesses * 10000;
+
if (prev && prev->ar.end == r->ar.start &&
abs(prev->nr_accesses - r->nr_accesses) <= thres &&
damon_sz_region(prev) + damon_sz_region(r) <= sz_limit)
@@ -1186,6 +1191,7 @@ static void damon_split_region_at(struct damon_target *t,
new->age = r->age;
new->last_nr_accesses = r->last_nr_accesses;
+ new->nr_accesses_bp = r->nr_accesses_bp;
damon_insert_region(new, r, damon_next_region(r), t);
}
--
2.25.1
Let nr_accesses_bp be calculated as a pseudo-moving sum that updated for
every sampling interval, using damon_moving_sum(). This is assumed to
be useful for cases that the aggregation interval is set quite huge, but
the monivoting results need to be collected earlier than next
aggregation interval is passed.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/damon.h | 12 +++++++++---
mm/damon/core.c | 16 +++++++++++++++-
mm/damon/paddr.c | 9 +++++----
mm/damon/vaddr.c | 12 +++++++-----
4 files changed, 36 insertions(+), 13 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 15f24b23c9a0..0fe13482df63 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -40,7 +40,8 @@ struct damon_addr_range {
* @ar: The address range of the region.
* @sampling_addr: Address of the sample for the next access check.
* @nr_accesses: Access frequency of this region.
- * @nr_accesses_bp: @nr_accesses in basis point (0.01%).
+ * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
+ * each sampling interval.
* @list: List head for siblings.
* @age: Age of this region.
*
@@ -51,7 +52,11 @@ struct damon_addr_range {
* damon_update_region_access_rate().
*
* @nr_accesses_bp is another representation of @nr_accesses in basis point
- * (1 in 10,000) that updated every aggregation interval.
+ * (1 in 10,000) that updated for every &damon_attrs->sample_interval in a
+ * manner similar to moving sum. By the algorithm, this value becomes
+ * @nr_accesses * 10000 for every &struct damon_attrs->aggr_interval. This can
+ * be used when the aggregation interval is too huge and therefore cannot wait
+ * for it before getting the access monitoring results.
*
* @age is initially zero, increased for each aggregation interval, and reset
* to zero again if the access frequency is significantly changed. If two
@@ -629,7 +634,8 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
unsigned int nr_ranges);
unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
unsigned int len_window, unsigned int new_value);
-void damon_update_region_access_rate(struct damon_region *r, bool accessed);
+void damon_update_region_access_rate(struct damon_region *r, bool accessed,
+ struct damon_attrs *attrs);
struct damos_filter *damos_new_filter(enum damos_filter_type type,
bool matching);
diff --git a/mm/damon/core.c b/mm/damon/core.c
index ce85c00b0a4c..29ee1fc18393 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -1599,14 +1599,28 @@ unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
* damon_update_region_access_rate() - Update the access rate of a region.
* @r: The DAMON region to update for its access check result.
* @accessed: Whether the region has accessed during last sampling interval.
+ * @attrs: The damon_attrs of the DAMON context.
*
* Update the access rate of a region with the region's last sampling interval
* access check result.
*
* Usually this will be called by &damon_operations->check_accesses callback.
*/
-void damon_update_region_access_rate(struct damon_region *r, bool accessed)
+void damon_update_region_access_rate(struct damon_region *r, bool accessed,
+ struct damon_attrs *attrs)
{
+ unsigned int len_window = 1;
+
+ /*
+ * sample_interval can be zero, but cannot be larger than
+ * aggr_interval, owing to validation of damon_set_attrs().
+ */
+ if (attrs->sample_interval)
+ len_window = attrs->aggr_interval / attrs->sample_interval;
+ r->nr_accesses_bp = damon_moving_sum(r->nr_accesses_bp,
+ r->last_nr_accesses * 10000, len_window,
+ accessed ? 10000 : 0);
+
if (accessed)
r->nr_accesses++;
}
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 44f21860b555..081e2a325778 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -148,7 +148,8 @@ static bool damon_pa_young(unsigned long paddr, unsigned long *folio_sz)
return accessed;
}
-static void __damon_pa_check_access(struct damon_region *r)
+static void __damon_pa_check_access(struct damon_region *r,
+ struct damon_attrs *attrs)
{
static unsigned long last_addr;
static unsigned long last_folio_sz = PAGE_SIZE;
@@ -157,12 +158,12 @@ static void __damon_pa_check_access(struct damon_region *r)
/* If the region is in the last checked page, reuse the result */
if (ALIGN_DOWN(last_addr, last_folio_sz) ==
ALIGN_DOWN(r->sampling_addr, last_folio_sz)) {
- damon_update_region_access_rate(r, last_accessed);
+ damon_update_region_access_rate(r, last_accessed, attrs);
return;
}
last_accessed = damon_pa_young(r->sampling_addr, &last_folio_sz);
- damon_update_region_access_rate(r, last_accessed);
+ damon_update_region_access_rate(r, last_accessed, attrs);
last_addr = r->sampling_addr;
}
@@ -175,7 +176,7 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx)
damon_for_each_target(t, ctx) {
damon_for_each_region(r, t) {
- __damon_pa_check_access(r);
+ __damon_pa_check_access(r, &ctx->attrs);
max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
}
}
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index e36303271f9d..af2cb82e1fad 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -557,26 +557,27 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
* r the region to be checked
*/
static void __damon_va_check_access(struct mm_struct *mm,
- struct damon_region *r, bool same_target)
+ struct damon_region *r, bool same_target,
+ struct damon_attrs *attrs)
{
static unsigned long last_addr;
static unsigned long last_folio_sz = PAGE_SIZE;
static bool last_accessed;
if (!mm) {
- damon_update_region_access_rate(r, false);
+ damon_update_region_access_rate(r, false, attrs);
return;
}
/* If the region is in the last checked page, reuse the result */
if (same_target && (ALIGN_DOWN(last_addr, last_folio_sz) ==
ALIGN_DOWN(r->sampling_addr, last_folio_sz))) {
- damon_update_region_access_rate(r, last_accessed);
+ damon_update_region_access_rate(r, last_accessed, attrs);
return;
}
last_accessed = damon_va_young(mm, r->sampling_addr, &last_folio_sz);
- damon_update_region_access_rate(r, last_accessed);
+ damon_update_region_access_rate(r, last_accessed, attrs);
last_addr = r->sampling_addr;
}
@@ -593,7 +594,8 @@ static unsigned int damon_va_check_accesses(struct damon_ctx *ctx)
mm = damon_get_mm(t);
same_target = false;
damon_for_each_region(r, t) {
- __damon_va_check_access(mm, r, same_target);
+ __damon_va_check_access(mm, r, same_target,
+ &ctx->attrs);
max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
same_target = true;
}
--
2.25.1
damon_merge_regions_of(), which is called for each aggregation interval,
updates nr_accesses_bp to nr_accesses * 10000. However, nr_accesses_bp
is updated for each sampling interval via damon_moving_sum() using the
aggregation interval as the moving time window. And by the definition
of the algorithm, the value becomes same to discrete-window based sum
for each time window-aligned time. Hence, nr_accesses_bp will be same
to nr_accesses * 10000 for each aggregation interval without explicit
update. Remove the unnecessary update of nr_accesses_bp in
damon_merge_regions_of().
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/core.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 29ee1fc18393..45cc108c0fe1 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -1141,8 +1141,6 @@ static void damon_merge_regions_of(struct damon_target *t, unsigned int thres,
else
r->age++;
- r->nr_accesses_bp = r->nr_accesses * 10000;
-
if (prev && prev->ar.end == r->ar.start &&
abs(prev->nr_accesses - r->nr_accesses) <= thres &&
damon_sz_region(prev) + damon_sz_region(r) <= sz_limit)
--
2.25.1