From: SeongJae Park <[email protected]>
DAMON[1] can be used as a primitive for data access awared memory management
optimizations. That said, users who want such optimizations should run DAMON,
read the monitoring results, analyze it, plan a new memory management scheme,
and apply the new scheme by themselves. Such efforts will be inevitable for
some complicated optimizations.
However, in many other cases, the users could simply want the system to apply a
memory management action to a memory region of a specific size having a
specific access frequency for a specific time. For example, "page out a memory
region larger than 100 MiB keeping only rare accesses more than 10 minutes", or
"Use THP for a memory region larger than 2 MiB continuously accessed for more
than 1 seconds".
This RFC patchset makes DAMON to handle such data access monitoring-based
operation schemes. With this change, users can do the data access awared
optimizations by simply specifying their schemes to DAMON.
Evaluations
===========
Transparent Huge Pages (THP) subsystem could waste memory space in some cases
because it aggressively promotes regular pages to huge pages. For the reason,
use of THP is prohivited by a number of memory intensive programs such as
Redis[1] and MongoDB[2].
Below two simple data access monitoring-based operation schemes might be
helpful for the problem:
# format: <min/max size> <min/max frequency (0-100)> <min/max age> <action>
# If a memory region larger than 2 MiB is showing access rate higher than
# 5% for more than 1 second, apply MADV_HUGEPAGE to the region.
2M null 5 null 1s null hugepage
# If a memory region larger than 2 MiB is showing access rate lower than 5%
# for more than 1 second, apply MADV_NOHUGEPAGE to the region.
2M null null 5 1s null nohugepage
We can expect the schmes might reduce the memory space overhead but preserve
some amount of the THP's performance benefit.
Please note that I made these schemes with only my straightforward instinction.
Therefore, these might not be optimized schemes.
Setup
-----
On a QEMU/KVM based virtual machine on an Intel i7 host machine running Ubuntu
18.04, I measure runtime and memory usage of various realistic workloads with
several configurations. I use 14 and 13 workloads in PARSEC3[3] and
SPLASH-2X[4] benchmark suites, respectively. I personally use another wrapper
scripts[5] for setup and run of the workloads.
For the measurement of memory usage, we drop caches
before starting each of the workloads and monitor 'MemFree' in the
'/proc/meminfo' file.
The configurations are as below:
orig: Linux v5.5 with 'madvise' THP policy
thp: Linux v5.5 with 'always' THP policy
ethp: Linux v5.5 applying the above schemes
[1] "Redis latency problems troubleshooting", https://redis.io/topics/latency
[2] "Disable Transparent Huge Pages (THP)",
https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
[3] "The PARSEC Becnhmark Suite", https://parsec.cs.princeton.edu/index.htm
[4] "SPLASH-2x", https://parsec.cs.princeton.edu/parsec3-doc.htm#splash2x
[5] "parsec3_on_ubuntu", https://github.com/sjp38/parsec3_on_ubuntu
Results
-------
Following sections show the results of the measurements. For brevity, I show
only memory space and runtime overheads of thp and ethp in percentage. For
instance, memory space overhead 76.94 of 'thp' for 'splash2x/ocean_ncp' means
the memory usage of 'splash2x/ocean_ncp' measured under 'thp' configuration was
176.94% of that measured under 'orig' configuration.
Note that the numbers are collected from only one measurement. Thus, below
numbers might contain some measurement errors. I will repeat the evaluations
several times and update the numbers with averages and stdevs, as soon as
prepared.
Memory Space Overheads
~~~~~~~~~~~~~~~~~~~~~~
Below shows measured memory space overheads (%) of 'thp' and 'ethp' compared to
'orig'.
workload thp ethp
parsec3 blackscholes 0.41 0.23
bodytrack -0.11 0.12
canneal -0.73 0.01
dedup 5.41 5.80
facesim 0.76 0.33
ferret -1.42 -0.24
fluidanimate -0.17 0.82
freqmine 0.03 0.17
raytrace 0.19 -0.55
streamcluster 1.16 1.65
swaptions 6.14 21.84
vips 1.63 1.65
x264 3.21 1.86
PARSEC3/AVG 1.63 1.48
splash2 barnes 0.18 1.82
fft 0.06 1.43
lu_cb 0.94 0.09
lu_ncb 1.09 0.43
ocean_cp 1.11 0.23
ocean_ncp 76.94 -0.08
radiosity 0.62 -0.05
radix -18.37 0.28
raytrace 14.52 3.64
volrend -0.69 1.48
water_nsquared -0.68 4.26
water_spatial 0.11 0.68
SPLASH2X/AVG 12.72 0.72
Averaged memory space overhead
PARSEC3: 1.62% (thp), 1.47% (ethp). ethp shows 1.10x lower overhead.
SPLASH2X: 12.717% (thp), 0.71% (ethp): ethp shows 18x lower overhead.
Best case: splash2x/ocean_ncp
Overheads: 76.94% (thp), -0.07% (ethp): ethp shows about -1099x lower overhead.
Apparently memory intensive workload, as it uses about 3.9 GiB memory in
'orig'.
Worst case: parsec3/swaptions
Overheads: 6.14% (thp), 21.84% (ethp): ethp shows 3.55x higher overhead.
Not memory-intensive workload, as it uses only 19 MiB memory in 'orig', though.
Runtime Overheads
~~~~~~~~~~~~~~~~~
Below shows measured runtime overheads (%) of 'thp' and 'ethp' compared to
'orig'.
workload thp ethp
parsec3 blackscholes -0.29 0.60
bodytrack -0.25 0.61
canneal -18.93 -15.60
dedup -1.79 -0.84
facesim -1.69 3.36
ferret -0.28 1.18
fluidanimate -1.08 2.89
freqmine 0.06 2.13
raytrace -1.35 0.79
streamcluster -13.69 -0.62
swaptions -1.83 -1.72
vips -2.05 -0.65
x264 14.96 -3.11
PARSEC3/AVG -3.82 -0.33
splash2 barnes -2.31 -0.95
fft -1.21 0.04
lu_cb -1.12 -0.14
lu_ncb -1.19 -0.24
ocean_cp -1.19 -0.48
ocean_ncp -1.29 -0.68
radiosity -1.29 -0.80
radix -0.33 -0.81
raytrace -0.22 -0.74
volrend -0.08 -0.75
water_nsquared -1.23 -0.57
water_spatial -1.16 -0.54
SPLASH2X/AVG -1.07 -0.51
Averaged runtime overhead
PARSEC3: -3.81% (thp), -0.38% (ethp): ethp preserves about 10% of THP speedup.
SPLASH2X: -1.07% (thp), -0.51% (ethp): ethp preserves about 50% of THP speedup.
Best case: parsec3/canneal
The overhead: -18.93% (thp), -15.60% (ethp): ethp preserves about 82% of THP
speedup.
Apparently memory intensive workload, as it uses about 1 GiB memory in average.
Worst case: parsec3/streamcluster
Seems memory-intensive workload, though it uses about 128 MiB memory.
The overheads: -13.69% (thp), -0.62% (ethp): ethp preserves only about 4% of
THP speedup.
In short, the straightforward data access monitoring-based operation scheme,
ethp, reduces memory space waste (1.10x lower for PARSEC3 and 18x lower for
SPLASH-2X) while preserving some amount of the THP's performance benefit (10%
for PARSEC3 and 50% for SPLASH-2X), as expected.
Sequence Of Patches
===================
The patches are based on the v5.5 plus v5 DAMON patchset[1] and Minchan's
``madvise()`` factor-out patch[2]. Minchan's patch was necessary for reuse of
``madvise()`` code in DAMON. You can also clone the complete git tree:
$ git clone git://github.com/sjp38/linux -b damos/rfc/v3
The web is also available:
https://github.com/sjp38/linux/releases/tag/damos/rfc/v3
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
The first patch allows DAMON to reuse ``madvise()`` code for the actions. The
second patch accounts age of each region. The third patch implements the
handling of the schemes in DAMON and exports a kernel space programming
interface for it. The fourth patch implements a debugfs interface for
privileged people and programs. The fifth and sixth patches each adds
kunittests and selftests for these changes, and finally the seventhe patch
modifies the user space tool for DAMON to support description and applying of
schemes in human freiendly way.
Patch History
=============
Changes from RFC v2
(https://lore.kernel.org/linux-mm/[email protected]/)
- Fix aging mechanism for more better 'old region' selection
- Add more kunittests and kselftests for this patchset
- Support more human friedly description and application of 'schemes'
Changes from RFC v1
(https://lore.kernel.org/linux-mm/[email protected]/)
- Properly adjust age accounting related properties after splitting, merging,
and action applying
SeongJae Park (7):
mm/madvise: Export madvise_common() to mm internal code
mm/damon: Account age of target regions
mm/damon: Implement data access monitoring-based operation schemes
mm/damon/schemes: Implement a debugfs interface
mm/damon-test: Add kunit test case for regions age accounting
mm/damon/selftests: Add 'schemes' debugfs tests
damon/tools: Support more human friendly 'schemes' control
include/linux/damon.h | 29 ++
mm/damon-test.h | 5 +
mm/damon.c | 414 +++++++++++++++++-
mm/internal.h | 4 +
mm/madvise.c | 3 +-
tools/damon/_convert_damos.py | 125 ++++++
tools/damon/_damon.py | 143 ++++++
tools/damon/damo | 7 +
tools/damon/record.py | 133 +-----
tools/damon/schemes.py | 105 +++++
.../testing/selftests/damon/debugfs_attrs.sh | 29 ++
11 files changed, 867 insertions(+), 130 deletions(-)
create mode 100755 tools/damon/_convert_damos.py
create mode 100644 tools/damon/_damon.py
create mode 100644 tools/damon/schemes.py
--
2.17.1
From: SeongJae Park <[email protected]>
In many cases, users might use DAMON for simple data access awared
memory management optimizations such as applying an operation scheme to
a memory region of a specific size having a specific access frequency
for a specific time. For example, "page out a memory region larger than
100 MiB but having a low access frequency more than 10 minutes", or "Use
THP for a memory region larger than 2 MiB having a high access frequency
for more than 2 seconds".
To minimize users from spending their time for implementation of such
simple data access monitoring-based operation schemes, this commit makes
DAMON to handle such schemes directly. With this commit, users can
simply specify their desired schemes to DAMON.
Each of the schemes is composed with conditions for filtering of the
target memory regions and desired memory management action for the
target. In specific, the format is::
<min/max size> <min/max access frequency> <min/max age> <action>
The filtering conditions are size of memory region, number of accesses
to the region monitored by DAMON, and the age of the region. The age of
region is incremented periodically but reset when its addresses or
access frequency has significanly changed or the action of a scheme has
applied. For the action, current implementation supports only a few of
madvise() hints, ``MADV_WILLNEED``, ``MADV_COLD``, ``MADV_PAGEOUT``,
``MADV_HUGEPAGE``, and ``MADV_NOHUGEPAGE``.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/damon.h | 24 ++++++++
mm/damon.c | 134 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 158 insertions(+)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 50fbe308590e..8cb2452579ee 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -36,6 +36,27 @@ struct damon_task {
struct list_head list;
};
+/* Data Access Monitoring-based Operation Scheme */
+enum damos_action {
+ DAMOS_WILLNEED,
+ DAMOS_COLD,
+ DAMOS_PAGEOUT,
+ DAMOS_HUGEPAGE,
+ DAMOS_NOHUGEPAGE,
+ DAMOS_ACTION_LEN,
+};
+
+struct damos {
+ unsigned int min_sz_region;
+ unsigned int max_sz_region;
+ unsigned int min_nr_accesses;
+ unsigned int max_nr_accesses;
+ unsigned int min_age_region;
+ unsigned int max_age_region;
+ enum damos_action action;
+ struct list_head list;
+};
+
struct damon_ctx {
unsigned long sample_interval;
unsigned long aggr_interval;
@@ -58,6 +79,7 @@ struct damon_ctx {
struct rnd_state rndseed;
struct list_head tasks_list; /* 'damon_task' objects */
+ struct list_head schemes_list; /* 'damos' objects */
/* callbacks */
void (*sample_cb)(struct damon_ctx *context);
@@ -66,6 +88,8 @@ struct damon_ctx {
int damon_set_pids(struct damon_ctx *ctx,
unsigned long *pids, ssize_t nr_pids);
+int damon_set_schemes(struct damon_ctx *ctx,
+ struct damos **schemes, ssize_t nr_schemes);
int damon_set_recording(struct damon_ctx *ctx,
unsigned int rbuf_len, char *rfile_path);
int damon_set_attrs(struct damon_ctx *ctx, unsigned long s, unsigned long a,
diff --git a/mm/damon.c b/mm/damon.c
index e08dba2e2717..a4d6dff60b1f 100644
--- a/mm/damon.c
+++ b/mm/damon.c
@@ -11,6 +11,7 @@
#define CREATE_TRACE_POINTS
+#include <asm-generic/mman-common.h>
#include <linux/damon.h>
#include <linux/debugfs.h>
#include <linux/delay.h>
@@ -24,6 +25,8 @@
#include <linux/slab.h>
#include <trace/events/damon.h>
+#include "internal.h"
+
#define damon_get_task_struct(t) \
(get_pid_task(find_vpid(t->pid), PIDTYPE_PID))
@@ -45,6 +48,12 @@
#define damon_for_each_task_safe(ctx, t, next) \
list_for_each_entry_safe(t, next, &(ctx)->tasks_list, list)
+#define damon_for_each_schemes(ctx, r) \
+ list_for_each_entry(r, &(ctx)->schemes_list, list)
+
+#define damon_for_each_schemes_safe(ctx, s, next) \
+ list_for_each_entry_safe(s, next, &(ctx)->schemes_list, list)
+
#define MAX_RFILE_PATH_LEN 256
/* Get a random number in [l, r) */
@@ -190,6 +199,27 @@ static void damon_destroy_task(struct damon_task *t)
damon_free_task(t);
}
+static void damon_add_scheme(struct damon_ctx *ctx, struct damos *s)
+{
+ list_add_tail(&s->list, &ctx->schemes_list);
+}
+
+static void damon_del_scheme(struct damos *s)
+{
+ list_del(&s->list);
+}
+
+static void damon_free_scheme(struct damos *s)
+{
+ kfree(s);
+}
+
+static void damon_destroy_scheme(struct damos *s)
+{
+ damon_del_scheme(s);
+ damon_free_scheme(s);
+}
+
/*
* Returns number of monitoring target tasks
*/
@@ -642,6 +672,87 @@ static void kdamond_count_age(struct damon_ctx *c, unsigned int threshold)
}
}
+static int damos_do_action(struct damon_task *task, struct damon_region *r,
+ enum damos_action action)
+{
+ struct task_struct *t;
+ struct mm_struct *mm;
+ int madv_action;
+ int ret = -EINVAL;
+
+ switch (action) {
+ case DAMOS_WILLNEED:
+ madv_action = MADV_WILLNEED;
+ break;
+ case DAMOS_COLD:
+ madv_action = MADV_COLD;
+ break;
+ case DAMOS_PAGEOUT:
+ madv_action = MADV_PAGEOUT;
+ break;
+ case DAMOS_HUGEPAGE:
+ madv_action = MADV_HUGEPAGE;
+ break;
+ case DAMOS_NOHUGEPAGE:
+ madv_action = MADV_NOHUGEPAGE;
+ break;
+ default:
+ pr_warn("Wrong action %d\n", action);
+ goto out;
+ }
+
+ t = damon_get_task_struct(task);
+ if (!t)
+ goto out;
+ mm = damon_get_mm(task);
+ if (!mm)
+ goto put_task_out;
+
+ ret = madvise_common(t, mm, PAGE_ALIGN(r->vm_start),
+ PAGE_ALIGN(r->vm_end - r->vm_start), madv_action);
+
+ mmput(mm);
+put_task_out:
+ put_task_struct(t);
+out:
+ return ret;
+}
+
+static void damon_do_apply_schemes(struct damon_ctx *c, struct damon_task *t,
+ struct damon_region *r)
+{
+ struct damos *s;
+ unsigned long sz;
+
+ damon_for_each_schemes(c, s) {
+ sz = r->vm_end - r->vm_start;
+ if ((s->min_sz_region && sz < s->min_sz_region) ||
+ (s->max_sz_region && s->max_sz_region < sz))
+ continue;
+ if ((s->min_nr_accesses && r->nr_accesses < s->min_nr_accesses)
+ || (s->max_nr_accesses &&
+ s->max_nr_accesses < r->nr_accesses))
+ continue;
+ if ((s->min_age_region && r->age < s->min_age_region) ||
+ (s->max_age_region &&
+ s->max_age_region < r->age))
+ continue;
+ damos_do_action(t, r, s->action);
+ r->age = 0;
+ }
+}
+
+static void kdamond_apply_schemes(struct damon_ctx *c)
+{
+ struct damon_task *t;
+ struct damon_region *r;
+
+ damon_for_each_task(c, t) {
+ damon_for_each_region(r, t)
+ damon_do_apply_schemes(c, t, r);
+ }
+}
+
#define sz_damon_region(r) (r->vm_end - r->vm_start)
/*
@@ -966,6 +1077,7 @@ static int kdamond_fn(void *data)
kdamond_count_age(ctx, max_nr_accesses / 10);
if (ctx->aggregate_cb)
ctx->aggregate_cb(ctx);
+ kdamond_apply_schemes(ctx);
kdamond_flush_aggregated(ctx);
kdamond_split_regions(ctx);
}
@@ -1040,6 +1152,27 @@ int damon_stop(struct damon_ctx *ctx)
return damon_turn_kdamond(ctx, false);
}
+/*
+ * Set the data access monitoring oriented schemes
+ *
+ * NOTE: This function should not be called while the kdamond of the context is
+ * running.
+ *
+ * Returns 0 if success, or negative error code otherwise.
+ */
+int damon_set_schemes(struct damon_ctx *ctx, struct damos **schemes,
+ ssize_t nr_schemes)
+{
+ struct damos *s, *next;
+ ssize_t i;
+
+ damon_for_each_schemes_safe(ctx, s, next)
+ damon_destroy_scheme(s);
+ for (i = 0; i < nr_schemes; i++)
+ damon_add_scheme(ctx, schemes[i]);
+ return 0;
+}
+
/*
* This function should not be called while the kdamond is running.
*/
@@ -1485,6 +1618,7 @@ static int __init damon_init_user_ctx(void)
prandom_seed_state(&ctx->rndseed, 42);
INIT_LIST_HEAD(&ctx->tasks_list);
+ INIT_LIST_HEAD(&ctx->schemes_list);
ctx->sample_cb = NULL;
ctx->aggregate_cb = NULL;
--
2.17.1
From: SeongJae Park <[email protected]>
After merges of regions, each region should know their last shape in
proper way to measure the changes from the last modification and reset
the age if the changes are significant. This commit adds kunit test
cases checking whether the regions are knowing their last shape properly
after merges of regions.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon-test.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/mm/damon-test.h b/mm/damon-test.h
index c7dc21325c77..2ba757357211 100644
--- a/mm/damon-test.h
+++ b/mm/damon-test.h
@@ -540,6 +540,8 @@ static void damon_test_merge_regions_of(struct kunit *test)
unsigned long saddrs[] = {0, 114, 130, 156, 170};
unsigned long eaddrs[] = {112, 130, 156, 170, 230};
+ unsigned long lsa[] = {0, 114, 130, 156, 184};
+ unsigned long lea[] = {100, 122, 156, 170, 230};
int i;
t = damon_new_task(42);
@@ -556,6 +558,9 @@ static void damon_test_merge_regions_of(struct kunit *test)
r = damon_nth_region_of(t, i);
KUNIT_EXPECT_EQ(test, r->vm_start, saddrs[i]);
KUNIT_EXPECT_EQ(test, r->vm_end, eaddrs[i]);
+ KUNIT_EXPECT_EQ(test, r->last_vm_start, lsa[i]);
+ KUNIT_EXPECT_EQ(test, r->last_vm_end, lea[i]);
+
}
damon_free_task(t);
}
--
2.17.1
From: SeongJae Park <[email protected]>
This commit adds simple selftets for 'schemes' debugfs file of DAMON.
Signed-off-by: SeongJae Park <[email protected]>
---
.../testing/selftests/damon/debugfs_attrs.sh | 29 +++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/tools/testing/selftests/damon/debugfs_attrs.sh b/tools/testing/selftests/damon/debugfs_attrs.sh
index d5188b0f71b1..82a98c81975b 100755
--- a/tools/testing/selftests/damon/debugfs_attrs.sh
+++ b/tools/testing/selftests/damon/debugfs_attrs.sh
@@ -97,6 +97,35 @@ fi
echo $ORIG_CONTENT > $file
+# Test schemes file
+file="$DBGFS/schemes"
+
+ORIG_CONTENT=$(cat $file)
+echo "1 2 3 4 5 6 3" > $file
+if [ $? -ne 0 ]
+then
+ echo "$file write fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo "1 2
+3 4 5 6 3" > $file
+if [ $? -eq 0 ]
+then
+ echo "$file splitted write success (expected fail)"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo > $file
+if [ $? -ne 0 ]
+then
+ echo "$file empty string writing fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
# Test pids file
file="$DBGFS/pids"
--
2.17.1
From: SeongJae Park <[email protected]>
This commit implements a debugfs interface for the data access
monitoring oriented memory management schemes. It is supposed to be
used by administrators and/or privileged user space programs. Users can
read and update the rules using ``<debugfs>/damon/schemes`` file. The
format is::
<min/max size> <min/max access frequency> <min/max age> <action>
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 169 insertions(+), 2 deletions(-)
diff --git a/mm/damon.c b/mm/damon.c
index a4d6dff60b1f..b286372dbf0e 100644
--- a/mm/damon.c
+++ b/mm/damon.c
@@ -199,6 +199,29 @@ static void damon_destroy_task(struct damon_task *t)
damon_free_task(t);
}
+static struct damos *damon_new_scheme(
+ unsigned int min_sz_region, unsigned int max_sz_region,
+ unsigned int min_nr_accesses, unsigned int max_nr_accesses,
+ unsigned int min_age_region, unsigned int max_age_region,
+ enum damos_action action)
+{
+ struct damos *ret;
+
+ ret = kmalloc(sizeof(struct damos), GFP_KERNEL);
+ if (!ret)
+ return NULL;
+ ret->min_sz_region = min_sz_region;
+ ret->max_sz_region = max_sz_region;
+ ret->min_nr_accesses = min_nr_accesses;
+ ret->max_nr_accesses = max_nr_accesses;
+ ret->min_age_region = min_age_region;
+ ret->max_age_region = max_age_region;
+ ret->action = action;
+ INIT_LIST_HEAD(&ret->list);
+
+ return ret;
+}
+
static void damon_add_scheme(struct damon_ctx *ctx, struct damos *s)
{
list_add_tail(&s->list, &ctx->schemes_list);
@@ -1329,6 +1352,144 @@ static ssize_t debugfs_monitor_on_write(struct file *file,
return ret;
}
+static ssize_t sprint_schemes(struct damon_ctx *c, char *buf, ssize_t len)
+{
+ struct damos *s;
+ int written = 0;
+ int rc;
+
+ damon_for_each_schemes(c, s) {
+ rc = snprintf(&buf[written], len - written,
+ "%u %u %u %u %u %u %d\n",
+ s->min_sz_region, s->max_sz_region,
+ s->min_nr_accesses, s->max_nr_accesses,
+ s->min_age_region, s->max_age_region,
+ s->action);
+ if (!rc)
+ return -ENOMEM;
+ written += rc;
+ }
+ return written;
+}
+
+static ssize_t debugfs_schemes_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = &damon_user_ctx;
+ char *kbuf;
+ ssize_t ret;
+
+ kbuf = kmalloc(count, GFP_KERNEL);
+ if (!kbuf)
+ return -ENOMEM;
+
+ ret = sprint_schemes(ctx, kbuf, count);
+ if (ret < 0)
+ goto out;
+ ret = simple_read_from_buffer(buf, count, ppos, kbuf, ret);
+
+out:
+ kfree(kbuf);
+ return ret;
+}
+
+static void free_schemes_arr(struct damos **schemes, ssize_t nr_schemes)
+{
+ ssize_t i;
+
+ for (i = 0; i < nr_schemes; i++)
+ kfree(schemes[i]);
+ kfree(schemes);
+}
+
+/*
+ * Converts a string into an array of struct damos pointers
+ *
+ * Returns an array of struct damos pointers that converted if the conversion
+ * success, or NULL otherwise.
+ */
+static struct damos **str_to_schemes(const char *str, ssize_t len,
+ ssize_t *nr_schemes)
+{
+ struct damos *scheme, **schemes;
+ const int max_nr_schemes = 256;
+ int pos = 0, parsed, ret;
+ unsigned int min_sz, max_sz, min_nr_a, max_nr_a, min_age, max_age;
+ int action;
+
+ schemes = kmalloc_array(max_nr_schemes, sizeof(struct damos *),
+ GFP_KERNEL);
+ if (!schemes)
+ return NULL;
+
+ *nr_schemes = 0;
+ while (pos < len && *nr_schemes < max_nr_schemes) {
+ ret = sscanf(&str[pos], "%u %u %u %u %u %u %d%n",
+ &min_sz, &max_sz, &min_nr_a, &max_nr_a,
+ &min_age, &max_age, &action, &parsed);
+ pos += parsed;
+ if (ret != 7)
+ break;
+ if (action >= DAMOS_ACTION_LEN) {
+ pr_err("wrong action %d\n", action);
+ goto fail;
+ }
+
+ scheme = damon_new_scheme(min_sz, max_sz, min_nr_a, max_nr_a,
+ min_age, max_age, action);
+ if (!scheme)
+ goto fail;
+
+ schemes[*nr_schemes] = scheme;
+ *nr_schemes += 1;
+ }
+ if (!*nr_schemes)
+ goto fail;
+ return schemes;
+fail:
+ free_schemes_arr(schemes, *nr_schemes);
+ return NULL;
+}
+
+static ssize_t debugfs_schemes_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = &damon_user_ctx;
+ char *kbuf;
+ struct damos **schemes;
+ ssize_t nr_schemes = 0, ret;
+
+ if (*ppos)
+ return -EINVAL;
+
+ kbuf = kmalloc_array(count, sizeof(char), GFP_KERNEL);
+ if (!kbuf)
+ return -ENOMEM;
+
+ ret = simple_write_to_buffer(kbuf, count, ppos, buf, count);
+ if (ret < 0)
+ goto out;
+
+ schemes = str_to_schemes(kbuf, ret, &nr_schemes);
+
+ spin_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond)
+ goto monitor_running;
+
+ damon_set_schemes(ctx, schemes, nr_schemes);
+ spin_unlock(&ctx->kdamond_lock);
+ goto out;
+
+monitor_running:
+ spin_unlock(&ctx->kdamond_lock);
+ pr_err("%s: kdamond is running. Turn it off first.\n", __func__);
+ ret = -EINVAL;
+ free_schemes_arr(schemes, nr_schemes);
+out:
+ kfree(kbuf);
+ return ret;
+}
+
static ssize_t damon_sprint_pids(struct damon_ctx *ctx, char *buf, ssize_t len)
{
struct damon_task *t;
@@ -1559,6 +1720,12 @@ static const struct file_operations pids_fops = {
.write = debugfs_pids_write,
};
+static const struct file_operations schemes_fops = {
+ .owner = THIS_MODULE,
+ .read = debugfs_schemes_read,
+ .write = debugfs_schemes_write,
+};
+
static const struct file_operations record_fops = {
.owner = THIS_MODULE,
.read = debugfs_record_read,
@@ -1575,10 +1742,10 @@ static struct dentry *debugfs_root;
static int __init debugfs_init(void)
{
- const char * const file_names[] = {"attrs", "record",
+ const char * const file_names[] = {"attrs", "record", "schemes",
"pids", "monitor_on"};
const struct file_operations *fops[] = {&attrs_fops, &record_fops,
- &pids_fops, &monitor_on_fops};
+ &schemes_fops, &pids_fops, &monitor_on_fops};
int i;
debugfs_root = debugfs_create_dir("damon", NULL);
--
2.17.1
From: SeongJae Park <[email protected]>
This commit implements 'schemes' subcommand of the damon userspace tool.
It can be used to describe and apply the data access monitoring-based
operation schemes in more human friendly fashion.
Signed-off-by: SeongJae Park <[email protected]>
---
tools/damon/_convert_damos.py | 125 +++++++++++++++++++++++++++++
tools/damon/_damon.py | 143 ++++++++++++++++++++++++++++++++++
tools/damon/damo | 7 ++
tools/damon/record.py | 133 +++----------------------------
tools/damon/schemes.py | 105 +++++++++++++++++++++++++
5 files changed, 391 insertions(+), 122 deletions(-)
create mode 100755 tools/damon/_convert_damos.py
create mode 100644 tools/damon/_damon.py
create mode 100644 tools/damon/schemes.py
diff --git a/tools/damon/_convert_damos.py b/tools/damon/_convert_damos.py
new file mode 100755
index 000000000000..0f1e7e3d4ccc
--- /dev/null
+++ b/tools/damon/_convert_damos.py
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+
+"""
+Change human readable data access monitoring-based operation schemes to the low
+level input for the '<debugfs>/damon/schemes' file. Below is an example of the
+schemes written in the human readable format:
+
+# format is: <min/max size> <min/max frequency (0-100)> <min/max age> <action>
+# lines starts with '#' or blank are ignored.
+# B/K/M/G/T for Bytes/KiB/MiB/GiB/TiB
+# us/ms/s/m/h/d for micro-seconds/milli-seconds/seconds/minutes/hours/days
+# 'null' means zero, which passes the check
+
+# if a region (no matter of its size) keeps a high access frequency for more
+# than 100ms, put the region on the head of the LRU list (call madvise() with
+# MADV_WILLNEED).
+null null 80 null 100ms null willneed
+
+# if a region keeps a low access frequency for more than 100ms, put the
+# region on the tail of the LRU list (call madvise() with MADV_COLD).
+0B 0B 10 20 200ms 1h cold
+
+# if a region keeps a very low access frequency for more than 100ms, swap
+# out the region immediately (call madvise() with MADV_PAGEOUT).
+0B null 0 10 100ms 2h pageout
+
+# if a region of a size bigger than 2MiB keeps a very high access frequency
+# for more than 100ms, let the region to use huge pages (call madvise()
+# with MADV_HUGEPAGE).
+2M null 90 99 100ms 2h hugepage
+
+# If a regions of a size bigger than 2MiB keeps no high access frequency
+# for more than 100ms, avoid the region from using huge pages (call
+# madvise() with MADV_NOHUGEPAGE).
+2M null 0 25 100ms 2h nohugepage
+"""
+
+import argparse
+
+unit_to_bytes = {'B': 1, 'K': 1024, 'M': 1024 * 1024, 'G': 1024 * 1024 * 1024,
+ 'T': 1024 * 1024 * 1024 * 1024}
+
+def text_to_bytes(txt):
+ if txt == 'null':
+ return 0
+ unit = txt[-1]
+ number = int(txt[:-1])
+ return number * unit_to_bytes[unit]
+
+unit_to_usecs = {'us': 1, 'ms': 1000, 's': 1000 * 1000, 'm': 60 * 1000 * 1000,
+ 'h': 60 * 60 * 1000 * 1000, 'd': 24 * 60 * 60 * 1000 * 1000}
+
+def text_to_us(txt):
+ if txt == 'null':
+ return 0
+ unit = txt[-2:]
+ if unit in ['us', 'ms']:
+ number = int(txt[:-2])
+ else:
+ unit = txt[-1]
+ number = int(txt[:-1])
+ return number * unit_to_usecs[unit]
+
+damos_action_to_int = {'DAMOS_WILLNEED': 0, 'DAMOS_COLD': 1,
+ 'DAMOS_PAGEOUT': 2, 'DAMOS_HUGEPAGE': 3, 'DAMOS_NOHUGEPAGE': 4}
+
+def text_to_damos_action(txt):
+ return damos_action_to_int['DAMOS_' + txt.upper()]
+
+def text_to_nr_accesses(txt, max_nr_accesses):
+ if txt == 'null':
+ return 0
+ return int(int(txt) * max_nr_accesses / 100)
+
+def debugfs_scheme(line, sample_interval, aggr_interval):
+ fields = line.split()
+ if len(fields) != 7:
+ print('wrong input line: %s' % line)
+ exit(1)
+
+ limit_nr_accesses = aggr_interval / sample_interval
+ try:
+ min_sz = text_to_bytes(fields[0])
+ max_sz = text_to_bytes(fields[1])
+ min_nr_accesses = text_to_nr_accesses(fields[2], limit_nr_accesses)
+ max_nr_accesses = text_to_nr_accesses(fields[3], limit_nr_accesses)
+ min_age = text_to_us(fields[4]) / aggr_interval
+ max_age = text_to_us(fields[5]) / aggr_interval
+ action = text_to_damos_action(fields[6])
+ except:
+ print('wrong input field')
+ raise
+ return '%d\t%d\t%d\t%d\t%d\t%d\t%d' % (min_sz, max_sz, min_nr_accesses,
+ max_nr_accesses, min_age, max_age, action)
+
+def convert(schemes_file, sample_interval, aggr_interval):
+ lines = []
+ with open(schemes_file, 'r') as f:
+ for line in f:
+ if line.startswith('#'):
+ continue
+ line = line.strip()
+ if line == '':
+ continue
+ lines.append(debugfs_scheme(line, sample_interval, aggr_interval))
+ return '\n'.join(lines)
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('input', metavar='<file>',
+ help='input file describing the schemes')
+ parser.add_argument('-s', '--sample', metavar='<interval>', type=int,
+ default=5000, help='sampling interval (us)')
+ parser.add_argument('-a', '--aggr', metavar='<interval>', type=int,
+ default=100000, help='aggregation interval (us)')
+ args = parser.parse_args()
+
+ schemes_file = args.input
+ sample_interval = args.sample
+ aggr_interval = args.aggr
+
+ print(convert(schemes_file, sample_interval, aggr_interval))
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/damon/_damon.py b/tools/damon/_damon.py
new file mode 100644
index 000000000000..0a703ec7471a
--- /dev/null
+++ b/tools/damon/_damon.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Contains core functions for DAMON debugfs control.
+"""
+
+import os
+import subprocess
+
+debugfs_attrs = None
+debugfs_record = None
+debugfs_schemes = None
+debugfs_pids = None
+debugfs_monitor_on = None
+
+def set_target_pid(pid):
+ return subprocess.call('echo %s > %s' % (pid, debugfs_pids), shell=True,
+ executable='/bin/bash')
+
+def turn_damon(on_off):
+ return subprocess.call("echo %s > %s" % (on_off, debugfs_monitor_on),
+ shell=True, executable="/bin/bash")
+
+def is_damon_running():
+ with open(debugfs_monitor_on, 'r') as f:
+ return f.read().strip() == 'on'
+
+class Attrs:
+ sample_interval = None
+ aggr_interval = None
+ regions_update_interval = None
+ min_nr_regions = None
+ max_nr_regions = None
+ rbuf_len = None
+ rfile_path = None
+ schemes = None
+
+ def __init__(self, s, a, r, n, x, l, f, c):
+ self.sample_interval = s
+ self.aggr_interval = a
+ self.regions_update_interval = r
+ self.min_nr_regions = n
+ self.max_nr_regions = x
+ self.rbuf_len = l
+ self.rfile_path = f
+ self.schemes = c
+
+ def __str__(self):
+ return "%s %s %s %s %s %s %s\n%s" % (self.sample_interval,
+ self.aggr_interval, self.regions_update_interval,
+ self.min_nr_regions, self.max_nr_regions, self.rbuf_len,
+ self.rfile_path, self.schemes)
+
+ def attr_str(self):
+ return "%s %s %s %s %s " % (self.sample_interval, self.aggr_interval,
+ self.regions_update_interval, self.min_nr_regions,
+ self.max_nr_regions)
+
+ def record_str(self):
+ return '%s %s ' % (self.rbuf_len, self.rfile_path)
+
+ def apply(self):
+ ret = subprocess.call('echo %s > %s' % (self.attr_str(), debugfs_attrs),
+ shell=True, executable='/bin/bash')
+ if ret:
+ return ret
+ ret = subprocess.call('echo %s > %s' % (self.record_str(),
+ debugfs_record), shell=True, executable='/bin/bash')
+ if ret:
+ return ret
+ return subprocess.call('echo %s > %s' % (
+ self.schemes.replace('\n', ' '), debugfs_schemes), shell=True,
+ executable='/bin/bash')
+
+def current_attrs():
+ with open(debugfs_attrs, 'r') as f:
+ attrs = f.read().split()
+ attrs = [int(x) for x in attrs]
+
+ with open(debugfs_record, 'r') as f:
+ rattrs = f.read().split()
+ attrs.append(int(rattrs[0]))
+ attrs.append(rattrs[1])
+
+ with open(debugfs_schemes, 'r') as f:
+ schemes = f.read()
+ attrs.append(schemes)
+
+ return Attrs(*attrs)
+
+def chk_update_debugfs(debugfs):
+ global debugfs_attrs
+ global debugfs_record
+ global debugfs_schemes
+ global debugfs_pids
+ global debugfs_monitor_on
+
+ debugfs_damon = os.path.join(debugfs, 'damon')
+ debugfs_attrs = os.path.join(debugfs_damon, 'attrs')
+ debugfs_record = os.path.join(debugfs_damon, 'record')
+ debugfs_schemes = os.path.join(debugfs_damon, 'schemes')
+ debugfs_pids = os.path.join(debugfs_damon, 'pids')
+ debugfs_monitor_on = os.path.join(debugfs_damon, 'monitor_on')
+
+ if not os.path.isdir(debugfs_damon):
+ print("damon debugfs dir (%s) not found", debugfs_damon)
+ exit(1)
+
+ for f in [debugfs_attrs, debugfs_record, debugfs_schemes, debugfs_pids,
+ debugfs_monitor_on]:
+ if not os.path.isfile(f):
+ print("damon debugfs file (%s) not found" % f)
+ exit(1)
+
+def cmd_args_to_attrs(args):
+ "Generate attributes with specified arguments"
+ sample_interval = args.sample
+ aggr_interval = args.aggr
+ regions_update_interval = args.updr
+ min_nr_regions = args.minr
+ max_nr_regions = args.maxr
+ rbuf_len = args.rbuf
+ if not os.path.isabs(args.out):
+ args.out = os.path.join(os.getcwd(), args.out)
+ rfile_path = args.out
+ schemes = args.schemes
+ return Attrs(sample_interval, aggr_interval, regions_update_interval,
+ min_nr_regions, max_nr_regions, rbuf_len, rfile_path, schemes)
+
+def set_attrs_argparser(parser):
+ parser.add_argument('-d', '--debugfs', metavar='<debugfs>', type=str,
+ default='/sys/kernel/debug', help='debugfs mounted path')
+ parser.add_argument('-s', '--sample', metavar='<interval>', type=int,
+ default=5000, help='sampling interval')
+ parser.add_argument('-a', '--aggr', metavar='<interval>', type=int,
+ default=100000, help='aggregate interval')
+ parser.add_argument('-u', '--updr', metavar='<interval>', type=int,
+ default=1000000, help='regions update interval')
+ parser.add_argument('-n', '--minr', metavar='<# regions>', type=int,
+ default=10, help='minimal number of regions')
+ parser.add_argument('-m', '--maxr', metavar='<# regions>', type=int,
+ default=1000, help='maximum number of regions')
diff --git a/tools/damon/damo b/tools/damon/damo
index 58e1099ae5fc..ce7180069bef 100755
--- a/tools/damon/damo
+++ b/tools/damon/damo
@@ -5,6 +5,7 @@ import argparse
import record
import report
+import schemes
class SubCmdHelpFormatter(argparse.RawDescriptionHelpFormatter):
def _format_action(self, action):
@@ -25,6 +26,10 @@ parser_record = subparser.add_parser('record',
help='record data accesses of the given target processes')
record.set_argparser(parser_record)
+parser_schemes = subparser.add_parser('schemes',
+ help='apply operation schemes to the given target process')
+schemes.set_argparser(parser_schemes)
+
parser_report = subparser.add_parser('report',
help='report the recorded data accesses in the specified form')
report.set_argparser(parser_report)
@@ -33,5 +38,7 @@ args = parser.parse_args()
if args.command == 'record':
record.main(args)
+elif args.command == 'schemes':
+ schemes.main(args)
elif args.command == 'report':
report.main(args)
diff --git a/tools/damon/record.py b/tools/damon/record.py
index a547d479a103..86252590fef9 100644
--- a/tools/damon/record.py
+++ b/tools/damon/record.py
@@ -6,28 +6,12 @@ Record data access patterns of the target process.
"""
import argparse
-import copy
import os
import signal
import subprocess
import time
-debugfs_attrs = None
-debugfs_record = None
-debugfs_pids = None
-debugfs_monitor_on = None
-
-def set_target_pid(pid):
- return subprocess.call('echo %s > %s' % (pid, debugfs_pids), shell=True,
- executable='/bin/bash')
-
-def turn_damon(on_off):
- return subprocess.call("echo %s > %s" % (on_off, debugfs_monitor_on),
- shell=True, executable="/bin/bash")
-
-def is_damon_running():
- with open(debugfs_monitor_on, 'r') as f:
- return f.read().strip() == 'on'
+import _damon
def do_record(target, is_target_cmd, attrs, old_attrs):
if os.path.isfile(attrs.rfile_path):
@@ -36,92 +20,28 @@ def do_record(target, is_target_cmd, attrs, old_attrs):
if attrs.apply():
print('attributes (%s) failed to be applied' % attrs)
cleanup_exit(old_attrs, -1)
- print('# damon attrs: %s' % attrs)
+ print('# damon attrs: %s %s' % (attrs.attr_str(), attrs.record_str()))
if is_target_cmd:
p = subprocess.Popen(target, shell=True, executable='/bin/bash')
target = p.pid
- if set_target_pid(target):
+ if _damon.set_target_pid(target):
print('pid setting (%s) failed' % target)
cleanup_exit(old_attrs, -2)
- if turn_damon('on'):
+ if _damon.turn_damon('on'):
print('could not turn on damon' % target)
cleanup_exit(old_attrs, -3)
if is_target_cmd:
p.wait()
while True:
# damon will turn it off by itself if the target tasks are terminated.
- if not is_damon_running():
+ if not _damon.is_damon_running():
break
time.sleep(1)
cleanup_exit(old_attrs, 0)
-class Attrs:
- sample_interval = None
- aggr_interval = None
- regions_update_interval = None
- min_nr_regions = None
- max_nr_regions = None
- rbuf_len = None
- rfile_path = None
-
- def __init__(self, s, a, r, n, x, l, f):
- self.sample_interval = s
- self.aggr_interval = a
- self.regions_update_interval = r
- self.min_nr_regions = n
- self.max_nr_regions = x
- self.rbuf_len = l
- self.rfile_path = f
-
- def __str__(self):
- return "%s %s %s %s %s %s %s" % (self.sample_interval, self.aggr_interval,
- self.regions_update_interval, self.min_nr_regions,
- self.max_nr_regions, self.rbuf_len, self.rfile_path)
-
- def attr_str(self):
- return "%s %s %s %s %s " % (self.sample_interval, self.aggr_interval,
- self.regions_update_interval, self.min_nr_regions,
- self.max_nr_regions)
-
- def record_str(self):
- return '%s %s ' % (self.rbuf_len, self.rfile_path)
-
- def apply(self):
- ret = subprocess.call('echo %s > %s' % (self.attr_str(), debugfs_attrs),
- shell=True, executable='/bin/bash')
- if ret:
- return ret
- return subprocess.call('echo %s > %s' % (self.record_str(),
- debugfs_record), shell=True, executable='/bin/bash')
-
-def current_attrs():
- with open(debugfs_attrs, 'r') as f:
- attrs = f.read().split()
- attrs = [int(x) for x in attrs]
-
- with open(debugfs_record, 'r') as f:
- rattrs = f.read().split()
- attrs.append(int(rattrs[0]))
- attrs.append(rattrs[1])
- return Attrs(*attrs)
-
-def cmd_args_to_attrs(args):
- "Generate attributes with specified arguments"
- sample_interval = args.sample
- aggr_interval = args.aggr
- regions_update_interval = args.updr
- min_nr_regions = args.minr
- max_nr_regions = args.maxr
- rbuf_len = args.rbuf
- if not os.path.isabs(args.out):
- args.out = os.path.join(os.getcwd(), args.out)
- rfile_path = args.out
- return Attrs(sample_interval, aggr_interval, regions_update_interval,
- min_nr_regions, max_nr_regions, rbuf_len, rfile_path)
-
def cleanup_exit(orig_attrs, exit_code):
- if is_damon_running():
+ if _damon.is_damon_running():
if turn_damon('off'):
print('failed to turn damon off!')
if orig_attrs:
@@ -133,51 +53,19 @@ def sighandler(signum, frame):
print('\nsignal %s received' % signum)
cleanup_exit(orig_attrs, signum)
-def chk_update_debugfs(debugfs):
- global debugfs_attrs
- global debugfs_record
- global debugfs_pids
- global debugfs_monitor_on
-
- debugfs_damon = os.path.join(debugfs, 'damon')
- debugfs_attrs = os.path.join(debugfs_damon, 'attrs')
- debugfs_record = os.path.join(debugfs_damon, 'record')
- debugfs_pids = os.path.join(debugfs_damon, 'pids')
- debugfs_monitor_on = os.path.join(debugfs_damon, 'monitor_on')
-
- if not os.path.isdir(debugfs_damon):
- print("damon debugfs dir (%s) not found", debugfs_damon)
- exit(1)
-
- for f in [debugfs_attrs, debugfs_record, debugfs_pids, debugfs_monitor_on]:
- if not os.path.isfile(f):
- print("damon debugfs file (%s) not found" % f)
- exit(1)
-
def chk_permission():
if os.geteuid() != 0:
print("Run as root")
exit(1)
def set_argparser(parser):
+ _damon.set_attrs_argparser(parser)
parser.add_argument('target', type=str, metavar='<target>',
help='the target command or the pid to record')
- parser.add_argument('-s', '--sample', metavar='<interval>', type=int,
- default=5000, help='sampling interval')
- parser.add_argument('-a', '--aggr', metavar='<interval>', type=int,
- default=100000, help='aggregate interval')
- parser.add_argument('-u', '--updr', metavar='<interval>', type=int,
- default=1000000, help='regions update interval')
- parser.add_argument('-n', '--minr', metavar='<# regions>', type=int,
- default=10, help='minimal number of regions')
- parser.add_argument('-m', '--maxr', metavar='<# regions>', type=int,
- default=1000, help='maximum number of regions')
parser.add_argument('-l', '--rbuf', metavar='<len>', type=int,
default=1024*1024, help='length of record result buffer')
parser.add_argument('-o', '--out', metavar='<file path>', type=str,
default='damon.data', help='output file path')
- parser.add_argument('-d', '--debugfs', metavar='<debugfs>', type=str,
- default='/sys/kernel/debug', help='debugfs mounted path')
def main(args=None):
global orig_attrs
@@ -187,13 +75,14 @@ def main(args=None):
args = parser.parse_args()
chk_permission()
- chk_update_debugfs(args.debugfs)
+ _damon.chk_update_debugfs(args.debugfs)
signal.signal(signal.SIGINT, sighandler)
signal.signal(signal.SIGTERM, sighandler)
- orig_attrs = current_attrs()
+ orig_attrs = _damon.current_attrs()
- new_attrs = cmd_args_to_attrs(args)
+ args.schemes = ''
+ new_attrs = _damon.cmd_args_to_attrs(args)
target = args.target
target_fields = target.split()
diff --git a/tools/damon/schemes.py b/tools/damon/schemes.py
new file mode 100644
index 000000000000..408a73813234
--- /dev/null
+++ b/tools/damon/schemes.py
@@ -0,0 +1,105 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Apply given operation schemes to the target process.
+"""
+
+import argparse
+import os
+import signal
+import subprocess
+import time
+
+import _convert_damos
+import _damon
+
+def run_damon(target, is_target_cmd, attrs, old_attrs):
+ if os.path.isfile(attrs.rfile_path):
+ os.rename(attrs.rfile_path, attrs.rfile_path + '.old')
+
+ if attrs.apply():
+ print('attributes (%s) failed to be applied' % attrs)
+ cleanup_exit(old_attrs, -1)
+ print('# damon attrs: %s %s' % (attrs.attr_str(), attrs.record_str()))
+ for line in attrs.schemes.split('\n'):
+ print('# scheme: %s' % line)
+ if is_target_cmd:
+ p = subprocess.Popen(target, shell=True, executable='/bin/bash')
+ target = p.pid
+ if _damon.set_target_pid(target):
+ print('pid setting (%s) failed' % target)
+ cleanup_exit(old_attrs, -2)
+ if _damon.turn_damon('on'):
+ print('could not turn on damon' % target)
+ cleanup_exit(old_attrs, -3)
+ if is_target_cmd:
+ p.wait()
+ while True:
+ # damon will turn it off by itself if the target tasks are terminated.
+ if not _damon.is_damon_running():
+ break
+ time.sleep(1)
+
+ cleanup_exit(old_attrs, 0)
+
+def cleanup_exit(orig_attrs, exit_code):
+ if _damon.is_damon_running():
+ if turn_damon('off'):
+ print('failed to turn damon off!')
+ if orig_attrs:
+ if orig_attrs.apply():
+ print('original attributes (%s) restoration failed!' % orig_attrs)
+ exit(exit_code)
+
+def sighandler(signum, frame):
+ print('\nsignal %s received' % signum)
+ cleanup_exit(orig_attrs, signum)
+
+def chk_permission():
+ if os.geteuid() != 0:
+ print("Run as root")
+ exit(1)
+
+def set_argparser(parser):
+ _damon.set_attrs_argparser(parser)
+ parser.add_argument('target', type=str, metavar='<target>',
+ help='the target command or the pid to record')
+ parser.add_argument('-c', '--schemes', metavar='<file>', type=str,
+ default='damon.schemes',
+ help='data access monitoring-based operation schemes')
+
+def main(args=None):
+ global orig_attrs
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ chk_permission()
+ _damon.chk_update_debugfs(args.debugfs)
+
+ signal.signal(signal.SIGINT, sighandler)
+ signal.signal(signal.SIGTERM, sighandler)
+ orig_attrs = _damon.current_attrs()
+
+ args.rbuf = 0
+ args.out = 'null'
+ args.schemes = _convert_damos.convert(args.schemes, args.sample, args.aggr)
+ new_attrs = _damon.cmd_args_to_attrs(args)
+ target = args.target
+
+ target_fields = target.split()
+ if not subprocess.call('which %s > /dev/null' % target_fields[0],
+ shell=True, executable='/bin/bash'):
+ run_damon(target, True, new_attrs, orig_attrs)
+ else:
+ try:
+ pid = int(target)
+ except:
+ print('target \'%s\' is neither a command, nor a pid' % target)
+ exit(1)
+ run_damon(target, False, new_attrs, orig_attrs)
+
+if __name__ == '__main__':
+ main()
--
2.17.1
On Tue, Feb 25, 2020 at 2:25 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> After merges of regions, each region should know their last shape in
> proper way to measure the changes from the last modification and reset
> the age if the changes are significant. This commit adds kunit test
> cases checking whether the regions are knowing their last shape properly
> after merges of regions.
>
> Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Brendan Higgins <[email protected]>