From: SeongJae Park <[email protected]>
Changes from Previous Version (v22)
===================================
- Support arbitrary targets; now DAMON incurs only zero space overhead for page
granularity idleness monitoring
- Reorder patches for easier review (Shakeel Butt)
- Introduce arbitrary targets with sampling first, then the overhead-accuracy
control logic
- Introduce data structure manipulation functions when it really used.
- Call callbacks explicitly, without macro (Shakeel Butt)
- Rename DAMON_PRIMITIVES to DAMON_VADDR (Shakeel Butt)
- Remove 'page_idle_lock' patch (Shakeel Butt)
- Drop pidfd support in debugfs (Shakeel Butt)
Introduction
============
DAMON is a data access monitoring framework for the Linux kernel. The core
mechanisms of DAMON called 'region based sampling' and 'adaptive regions
adjustment' (refer to 'mechanisms.rst' in the 11th patch of this patchset for
the detail) make it
- accurate (The monitored information is useful for DRAM level memory
management. It might not appropriate for Cache-level accuracy, though.),
- light-weight (The monitoring overhead is low enough to be applied online
while making no impact on the performance of the target workloads.), and
- scalable (the upper-bound of the instrumentation overhead is controllable
regardless of the size of target workloads.).
Using this framework, therefore, several memory management mechanisms such as
reclamation and THP can be optimized to aware real data access patterns.
Experimental access pattern aware memory management optimization works that
incurring high instrumentation overhead will be able to have another try.
Though DAMON is for kernel subsystems, it can be easily exposed to the user
space by writing a DAMON-wrapper kernel subsystem. Then, user space users who
have some special workloads will be able to write personalized tools or
applications for deeper understanding and specialized optimizations of their
systems.
Long-term Plan
--------------
DAMON is a part of a project called Data Access-aware Operating System (DAOS).
As the name implies, I want to improve the performance and efficiency of
systems using fine-grained data access patterns. The optimizations are for
both kernel and user spaces. I will therefore modify or create kernel
subsystems, export some of those to user space and implement user space library
/ tools. Below shows the layers and components for the project.
---------------------------------------------------------------------------
Primitives: PTE Accessed bit, PG_idle, rmap, (Intel CMT), ...
Framework: DAMON
Features: DAMOS, virtual addr, physical addr, ...
Applications: DAMON-debugfs, (DARC), ...
^^^^^^^^^^^^^^^^^^^^^^^ KERNEL SPACE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Raw Interface: debugfs, (sysfs), (damonfs), tracepoints, (sys_damon), ...
vvvvvvvvvvvvvvvvvvvvvvv USER SPACE vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Library: (libdamon), ...
Tools: DAMO, (perf), ...
---------------------------------------------------------------------------
The components in parentheses or marked as '...' are not implemented yet but in
the future plan. IOW, those are the TODO tasks of DAOS project. For more
detail, please refer to the plans:
https://lore.kernel.org/linux-mm/[email protected]/
Evaluations
===========
We evaluated DAMON's overhead, monitoring quality and usefulness using 24
realistic workloads on my QEMU/KVM based virtual machine running a kernel that
v23 DAMON patchset is applied.
DAMON is lightweight. It increases system memory usage by 0.42% and slows
target workloads down by 0.39%.
DAMON is accurate and useful for memory management optimizations. An
experimental DAMON-based operation scheme for THP, 'ethp', removes 81.45% of
THP memory overheads while preserving 50.09% of THP speedup. Another
experimental DAMON-based 'proactive reclamation' implementation, 'prcl',
reduces 91.45% of residential sets and 22.91% of system memory footprint while
incurring only 2.43% runtime overhead in the best case (parsec3/freqmine).
NOTE that the experimental THP optimization and proactive reclamation are not
for production but only for proof of concepts.
Please refer to the official document[1] or "Documentation/admin-guide/mm: Add
a document for DAMON" patch in this patchset for detailed evaluation setup and
results.
[1] https://damonitor.github.io/doc/html/latest-damon/admin-guide/mm/damon/eval.html
Real-world User Story
=====================
In summary, DAMON has used on production systems and proved its usefulness.
DAMON as a profiler
-------------------
We analyzed characteristics of a large scale production systems of our
customers using DAMON. The systems utilize 70GB DRAM and 36 CPUs. From this,
we were able to find interesting things below.
There were obviously different access pattern under idle workload and active
workload. Under the idle workload, it accessed large memory regions with low
frequency, while the active workload accessed small memory regions with high
freuqnecy.
DAMON found a 7GB memory region that showing obviously high access frequency
under the active workload. We believe this is the performance-effective
working set and need to be protected.
There was a 4KB memory region that showing highest access frequency under not
only active but also idle workloads. We think this must be a hottest code
section like thing that should never be paged out.
For this analysis, DAMON used only 0.3-1% of single CPU time. Because we used
recording-based analysis, it consumed about 3-12 MB of disk space per 20
minutes. This is only small amount of disk space, but we can further reduce
the disk usage by using non-recording-based DAMON features. I'd like to argue
that only DAMON can do such detailed analysis (finding 4KB highest region in
70GB memory) with the light overhead.
DAMON as a system optimization tool
-----------------------------------
We also found below potential performance problems on the systems and made
DAMON-based solutions.
The system doesn't want to make the workload suffer from the page reclamation
and thus it utilizes enough DRAM but no swap device. However, we found the
system is actively reclaiming file-backed pages, because the system has
intensive file IO. The file IO turned out to be not performance critical for
the workload, but the customer wanted to ensure performance critical
file-backed pages like code section to not mistakenly be evicted.
Using direct IO should or `mlock()` would be a straightforward solution, but
modifying the user space code is not easy for the customer. Alternatively, we
could use DAMON-based operation scheme[1]. By using it, we can ask DAMON to
track access frequency of each region and make
'process_madvise(MADV_WILLNEED)[2]' call for regions having specific size and
access frequency for a time interval.
We also found the system is having high number of TLB misses. We tried
'always' THP enabled policy and it greatly reduced TLB misses, but the page
reclamation also been more frequent due to the THP internal fragmentation
caused memory bloat. We could try another DAMON-based operation scheme that
applies 'MADV_HUGEPAGE' to memory regions having >=2MB size and high access
frequency, while applying 'MADV_NOHUGEPAGE' to regions having <2MB size and low
access frequency.
We do not own the systems so we only reported the analysis results and possible
optimization solutions to the customers. The customers satisfied about the
analysis results and promised to try the optimization guides.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-api/[email protected]/
Comparison with Idle Page Tracking
==================================
Idle Page Tracking allows users to set and read idleness of pages using a
bitmap file which represents each page with each bit of the file. One
recommended usage of it is working set size detection. Users can do that by
1. find PFN of each page for workloads in interest,
2. set all the pages as idle by doing writes to the bitmap file,
3. wait until the workload accesses its working set, and
4. read the idleness of the pages again and count pages became not idle.
NOTE: While Idle Page Tracking is for user space users, DAMON is primarily
designed for kernel subsystems though it can easily exposed to the user space.
Hence, this section only assumes such user space use of DAMON.
For what use cases Idle Page Tracking would be better?
------------------------------------------------------
1. Flexible usecases other than hotness monitoring.
Because Idle Page Tracking allows users to control the primitive (Page
idleness) by themselves, Idle Page Tracking users can do anything they want.
Meanwhile, DAMON is primarily designed to monitor the hotness of each memory
region. For this, DAMON asks users to provide sampling interval and
aggregation interval. For the reason, there could be some use case that using
Idle Page Tracking is simpler.
2. Physical memory monitoring.
Idle Page Tracking receives PFN range as input, so natively supports physical
memory monitoring.
DAMON is designed to be extensible for multiple address spaces and use cases by
implementing and using primitives for the given use case. Therefore, by
theory, DAMON has no limitation in the type of target address space as long as
primitives for the given address space exists. However, the default primitives
introduced by this patchset supports only virtual address spaces.
Therefore, for physical memory monitoring, you should implement your own
primitives and use it, or simply use Idle Page Tracking.
Nonetheless, RFC patchsets[1] for the physical memory address space primitives
is already available. It also supports user memory same to Idle Page Tracking.
[1] https://lore.kernel.org/linux-mm/[email protected]/
For what use cases DAMON is better?
-----------------------------------
1. Hotness Monitoring.
Idle Page Tracking let users know only if a page frame is accessed or not. For
hotness check, the user should write more code and use more memory. DAMON do
that by itself.
2. Low Monitoring Overhead
DAMON receives user's monitoring request with one step and then provide the
results. So, roughly speaking, DAMON require only O(1) user/kernel context
switches.
In case of Idle Page Tracking, however, because the interface receives
contiguous page frames, the number of user/kernel context switches increases as
the monitoring target becomes complex and huge. As a result, the context
switch overhead could be not negligible.
Moreover, DAMON is born to handle with the monitoring overhead. Because the
core mechanism is pure logical, Idle Page Tracking users might be able to
implement the mechanism on thier own, but it would be time consuming and the
user/kernel context switching will still more frequent than that of DAMON.
Also, the kernel subsystems cannot use the logic in this case.
3. Page granularity working set size detection.
Until v22 of this patchset, this was categorized as the thing Idle Page
Tracking could do better, because DAMON basically maintains additional metadata
for each of the monitoring target regions. So, in the page granularity working
set size detection use case, DAMON would incur (number of monitoring target
pages * size of metadata) memory overhead. Size of the single metadata item is
about 54 bytes, so assuming 4KB pages, about 1.3% of monitoring target pages
will be additionally used.
All essential metadata for Idle Page Tracking are embedded in 'struct page' and
page table entries. Therefore, in this use case, only one counter variable for
working set size accounting is required if Idle Page Tracking is used.
There are more details to consider, but roughly speaking, this is true in most
cases.
However, the situation changed from v23. Now DAMON supports arbitrary types of
monitoring targets, which don't use the metadata. Using that, DAMON can do the
working set size detection with no additional space overhead but less
user-kernel context switch. A first draft for the implementation of monitoring
primitives for this usage is available in a DAMON development tree[1]. An RFC
patchset for it based on this patchset will also be available soon.
[1] https://github.com/sjp38/linux/tree/damon/pgidle_hack
4. More future usecases
While Idle Page Tracking has tight coupling with base primitives (PG_Idle and
page table Accessed bits), DAMON is designed to be extensible for many use
cases and address spaces. If you need some special address type or want to use
special h/w access check primitives, you can write your own primitives for that
and configure DAMON to use those. Therefore, if your use case could be changed
a lot in future, using DAMON could be better.
Can I use both Idle Page Tracking and DAMON?
--------------------------------------------
Yes, though using them concurrently for overlapping memory regions could result
in interference to each other. Nevertheless, such use case would be rare or
makes no sense at all. Even in the case, the noise would bot be really
significant. So, you can choose whatever you want depending on the
characteristics of your use cases.
More Information
================
We prepared a showcase web site[1] that you can get more information. There
are
- the official documentations[2],
- the heatmap format dynamic access pattern of various realistic workloads for
heap area[3], mmap()-ed area[4], and stack[5] area,
- the dynamic working set size distribution[6] and chronological working set
size changes[7], and
- the latest performance test results[8].
[1] https://damonitor.github.io/_index
[2] https://damonitor.github.io/doc/html/latest-damon
[3] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.0.png.html
[4] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html
[5] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.2.png.html
[6] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html
[7] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
[8] https://damonitor.github.io/test/result/perf/latest/html/index.html
Baseline and Complete Git Trees
===============================
The patches are based on the v5.10. You can also clone the complete git
tree:
$ git clone git://github.com/sjp38/linux -b damon/patches/v23
The web is also available:
https://github.com/sjp38/linux/releases/tag/damon/patches/v23
There are a couple of trees for entire DAMON patchset series. It includes
future features. The first one[1] contains the changes for latest release,
while the other one[2] contains the changes for next release.
[1] https://github.com/sjp38/linux/tree/damon/master
[2] https://github.com/sjp38/linux/tree/damon/next
Sequence Of Patches
===================
First three patches implement the core logics of DAMON. The 1st patch
introduces basic sampling based hotness monitoring for arbitrary types of
targets. Following two patches implement the core mechanisms for control of
overhead and accuracy, namely regions based sampling (patch 2) and adaptive
regions adjustment (patch 3).
Now the essential parts of DAMON is complete, but it cannot work unless someone
provides monitoring primitives for a specific use case. The following two
patches make it just work for virtual address spaces monitoring. The 4th patch
makes 'PG_idle' can be used by DAMON and the 5th patch implements the virtual
memory address space specific monitoring primitives using page table Accessed
bits and the 'PG_idle' page flag.
Now DAMON just works for virtual address space monitoring via the kernel space
api. To let the user space users can use DAMON, following six patches add
interfaces for them. The 6th patch adds a tracepoint for monitoring results.
The 7th patch implements a DAMON application kernel module, namely damon-dbgfs,
that simply wraps DAMON and exposes DAMON interface to the user space via the
debugfs interface. To let the user space get the monitoring results more
easily, the 8th patch implements a simple recording feature in 'damon-dbgfs'.
The 9th patch further exports pid of monitoring thread (kdamond) to user space
for easier cpu usage accounting, and the 10th patch makes the debugfs interface
to support multiple contexts. Then, the 11th patch implements a user space
tool to provide a minimal reference to the debugfs interface and for high level
use/tests of the DAMON.
Three patches for maintainability follows. The 12th patch adds documentations
for both the user space and the kernel space. The 13th patch provides unit
tests (based on the kunit) while the 14th patch adds user space tests (based on
the kselftest).
Finally, the last patch (15th) updates the MAINTAINERS file.
Patch History
=============
Changes from v22
(https://lore.kernel.org/linux-mm/[email protected]/)
- Support arbitrary targets; now DAMON incurs only zero space overhead for page
granularity idleness monitoring
- Reorder patches for easier review (Shakeel Butt)
- Introduce arbitrary targets with sampling first, then the overhead-accuracy
control logic
- Introduce data structure manipulation functions when it really used.
- Call callbacks explicitly, without macro (Shakeel Butt)
- Rename DAMON_PRIMITIVES to DAMON_VADDR (Shakeel Butt)
- Remove 'page_idle_lock' patch (Shakeel Butt)
- Drop pidfd support in debugfs (Shakeel Butt)
Changes from v21
(https://lore.kernel.org/linux-doc/[email protected]/)
- Fix build warnings and errors (kernel test robot)
- Fix a memory leak (kmemleak)
- Respect KUNIT_ALL_TESTS
- Rebase on v5.9
- Update the evaluation results
Changes from v20
(https://lore.kernel.org/linux-mm/[email protected]/)
- s/snprintf()/scnprintf() (Marco Elver)
- Support multiple contexts for user space users (Shakeel Butt)
- Export pid of monitoring thread to user space (Shakeel Butt)
- Let coexistable with Idle Page Tracking
- Place three parts of DAMON (core, primitives, and dbgfs) in different files
Changes from v19
(https://lore.kernel.org/linux-mm/[email protected]/)
- Place 'CREATE_TRACE_POINTS' after '#include' statements (Steven Rostedt)
- Support large record file (Alkaid)
- Place 'put_pid()' of virtual monitoring targets in 'cleanup' callback
- Avoid conflict between concurrent DAMON users
- Update evaluation result document
Changes from v18
(https://lore.kernel.org/linux-mm/[email protected]/)
- Drop loadable module support (Mike Rapoport)
- Select PAGE_EXTENSION if !64BIT for 'set_page_young()'
- Take care of the MMU notification subscribers (Shakeel Butt)
- Substitute 'struct damon_task' with 'struct damon_target' for better abstract
- Use 'struct pid' instead of 'pid_t' as the target (Shakeel Butt)
- Support pidfd from the debugfs interface (Shakeel Butt)
- Fix typos (Greg Thelen)
- Properly isolate DAMON from other pmd/pte Accessed bit users (Greg Thelen)
- Rebase on v5.8
Changes from v17
(https://lore.kernel.org/linux-mm/[email protected]/)
- Reorganize the doc and remove png blobs (Mike Rapoport)
- Wordsmith mechnisms doc and commit messages
- tools/wss: Set default working set access frequency threshold
- Avoid race in damon deamon start
Please refer to the v17 patchset to get older history.
SeongJae Park (15):
mm: Introduce Data Access MONitor (DAMON)
mm/damon/core: Implement region-based sampling
mm/damon: Adaptively adjust regions
mm/idle_page_tracking: Make PG_idle reusable
mm/damon: Implement primitives for the virtual memory address spaces
mm/damon: Add a tracepoint
mm/damon: Implement a debugfs-based user space interface
mm/damon/dbgfs: Implement recording feature
mm/damon/dbgfs: Export kdamond pid to the user space
mm/damon/dbgfs: Support multiple contexts
tools: Introduce a minimal user-space tool for DAMON
Documentation: Add documents for DAMON
mm/damon: Add kunit tests
mm/damon: Add user space selftests
MAINTAINERS: Update for DAMON
Documentation/admin-guide/mm/damon/guide.rst | 157 ++++
Documentation/admin-guide/mm/damon/index.rst | 15 +
Documentation/admin-guide/mm/damon/plans.rst | 29 +
Documentation/admin-guide/mm/damon/start.rst | 96 ++
Documentation/admin-guide/mm/damon/usage.rst | 302 ++++++
Documentation/admin-guide/mm/index.rst | 1 +
Documentation/vm/damon/api.rst | 20 +
Documentation/vm/damon/design.rst | 166 ++++
Documentation/vm/damon/eval.rst | 227 +++++
Documentation/vm/damon/faq.rst | 58 ++
Documentation/vm/damon/index.rst | 31 +
Documentation/vm/index.rst | 1 +
MAINTAINERS | 12 +
include/linux/damon.h | 293 ++++++
include/linux/page-flags.h | 4 +-
include/linux/page_ext.h | 2 +-
include/linux/page_idle.h | 6 +-
include/trace/events/damon.h | 43 +
include/trace/events/mmflags.h | 2 +-
mm/Kconfig | 10 +
mm/Makefile | 1 +
mm/damon/Kconfig | 69 ++
mm/damon/Makefile | 5 +
mm/damon/core-test.h | 253 +++++
mm/damon/core.c | 717 +++++++++++++++
mm/damon/dbgfs-test.h | 209 +++++
mm/damon/dbgfs.c | 867 ++++++++++++++++++
mm/damon/vaddr-test.h | 328 +++++++
mm/damon/vaddr.c | 586 ++++++++++++
mm/page_ext.c | 12 +-
mm/page_idle.c | 10 -
tools/damon/.gitignore | 1 +
tools/damon/_damon.py | 130 +++
tools/damon/_dist.py | 35 +
tools/damon/_recfile.py | 23 +
tools/damon/bin2txt.py | 67 ++
tools/damon/damo | 37 +
tools/damon/heats.py | 362 ++++++++
tools/damon/nr_regions.py | 91 ++
tools/damon/record.py | 135 +++
tools/damon/report.py | 45 +
tools/damon/wss.py | 100 ++
tools/testing/selftests/damon/Makefile | 7 +
.../selftests/damon/_chk_dependency.sh | 28 +
tools/testing/selftests/damon/_chk_record.py | 109 +++
.../testing/selftests/damon/debugfs_attrs.sh | 161 ++++
.../testing/selftests/damon/debugfs_record.sh | 50 +
47 files changed, 5895 insertions(+), 18 deletions(-)
create mode 100644 Documentation/admin-guide/mm/damon/guide.rst
create mode 100644 Documentation/admin-guide/mm/damon/index.rst
create mode 100644 Documentation/admin-guide/mm/damon/plans.rst
create mode 100644 Documentation/admin-guide/mm/damon/start.rst
create mode 100644 Documentation/admin-guide/mm/damon/usage.rst
create mode 100644 Documentation/vm/damon/api.rst
create mode 100644 Documentation/vm/damon/design.rst
create mode 100644 Documentation/vm/damon/eval.rst
create mode 100644 Documentation/vm/damon/faq.rst
create mode 100644 Documentation/vm/damon/index.rst
create mode 100644 include/linux/damon.h
create mode 100644 include/trace/events/damon.h
create mode 100644 mm/damon/Kconfig
create mode 100644 mm/damon/Makefile
create mode 100644 mm/damon/core-test.h
create mode 100644 mm/damon/core.c
create mode 100644 mm/damon/dbgfs-test.h
create mode 100644 mm/damon/dbgfs.c
create mode 100644 mm/damon/vaddr-test.h
create mode 100644 mm/damon/vaddr.c
create mode 100644 tools/damon/.gitignore
create mode 100644 tools/damon/_damon.py
create mode 100644 tools/damon/_dist.py
create mode 100644 tools/damon/_recfile.py
create mode 100644 tools/damon/bin2txt.py
create mode 100755 tools/damon/damo
create mode 100644 tools/damon/heats.py
create mode 100644 tools/damon/nr_regions.py
create mode 100644 tools/damon/record.py
create mode 100644 tools/damon/report.py
create mode 100644 tools/damon/wss.py
create mode 100644 tools/testing/selftests/damon/Makefile
create mode 100644 tools/testing/selftests/damon/_chk_dependency.sh
create mode 100644 tools/testing/selftests/damon/_chk_record.py
create mode 100755 tools/testing/selftests/damon/debugfs_attrs.sh
create mode 100755 tools/testing/selftests/damon/debugfs_record.sh
--
2.17.1
From: SeongJae Park <[email protected]>
To avoid the unbounded increase of the overhead, DAMON groups adjacent
pages that assumed to have the same access frequencies into a region.
As long as the assumption (pages in a region have the same access
frequencies) is kept, only one page in the region is required to be
checked. Thus, for each ``sampling interval``,
1. the 'prepare_access_checks' primitive picks one page in each region,
2. waits for one ``sampling interval``,
3. checks whether the page is accessed meanwhile, and
4. increases the access frequency of the region if so.
Therefore, the monitoring overhead is controllable by adjusting the
number of regions. DAMON allows both the underlying primitives and user
callbacks adjust regions for the trade-off. In other words, this commit
makes DAMON to use not only time-based sampling but also space-based
sampling.
This scheme, however, cannot preserve the quality of the output if the
assumption is not guaranteed. Next commit will address this problem.
Another problem of this region abstraction is additional memory space
overhead for the regions metadata. For example, suppose page
granularity monitoring that doesn't want to know fine-grained access
frequency but only if each page accessed or not. Then, we can do that
by directly resetting and reading the PG_Idle flags and/or the PTE
Accessed bits. The metadata for the region abstraction is only burden
in the case. For the reason, this commit makes DAMON to support the
user-defined arbitrary target, which could be stored in a void pointer
of the monitoring context with specific target type.
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Leonard Foerster <[email protected]>
---
include/linux/damon.h | 109 ++++++++++++++++++++++++++++++--
mm/damon/core.c | 142 +++++++++++++++++++++++++++++++++++++++++-
2 files changed, 243 insertions(+), 8 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 387fa4399fc8..7d4685adc8a9 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -12,6 +12,48 @@
#include <linux/time64.h>
#include <linux/types.h>
+/**
+ * struct damon_addr_range - Represents an address region of [@start, @end).
+ * @start: Start address of the region (inclusive).
+ * @end: End address of the region (exclusive).
+ */
+struct damon_addr_range {
+ unsigned long start;
+ unsigned long end;
+};
+
+/**
+ * struct damon_region - Represents a monitoring target region.
+ * @ar: The address range of the region.
+ * @sampling_addr: Address of the sample for the next access check.
+ * @nr_accesses: Access frequency of this region.
+ * @list: List head for siblings.
+ */
+struct damon_region {
+ struct damon_addr_range ar;
+ unsigned long sampling_addr;
+ unsigned int nr_accesses;
+ struct list_head list;
+};
+
+/**
+ * struct damon_target - Represents a monitoring target.
+ * @id: Unique identifier for this target.
+ * @regions_list: Head of the monitoring target regions of this target.
+ * @list: List head for siblings.
+ *
+ * Each monitoring context could have multiple targets. For example, a context
+ * for virtual memory address spaces could have multiple target processes. The
+ * @id of each target should be unique among the targets of the context. For
+ * example, in the virtual address monitoring context, it could be a pidfd or
+ * an address of an mm_struct.
+ */
+struct damon_target {
+ unsigned long id;
+ struct list_head regions_list;
+ struct list_head list;
+};
+
struct damon_ctx;
/**
@@ -36,7 +78,8 @@ struct damon_ctx;
*
* @init_target_regions should construct proper monitoring target regions and
* link those to the DAMON context struct. The regions should be defined by
- * user and saved in @damon_ctx.target.
+ * user and saved in @damon_ctx.arbitrary_target if @damon_ctx.target_type is
+ * &DAMON_ARBITRARY_TARGET. Otherwise, &struct damon_region should be used.
* @update_target_regions should update the monitoring target regions for
* current status.
* @prepare_access_checks should manipulate the monitoring regions to be
@@ -46,7 +89,8 @@ struct damon_ctx;
* @reset_aggregated should reset the access monitoring results that aggregated
* by @check_accesses.
* @target_valid should check whether the target is still valid for the
- * monitoring.
+ * monitoring. It receives &damon_ctx.arbitrary_target or &struct damon_target
+ * pointer depends on &damon_ctx.target_type.
* @cleanup is called from @kdamond just before its termination. After this
* call, only @kdamond_lock and @kdamond will be touched.
*/
@@ -91,6 +135,17 @@ struct damon_callback {
int (*before_terminate)(struct damon_ctx *context);
};
+/**
+ * enum damon_target_type - Represents the type of the monitoring target.
+ *
+ * @DAMON_REGION_SAMPLING_TARGET: Region based sampling target.
+ * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
+ */
+enum damon_target_type {
+ DAMON_REGION_SAMPLING_TARGET,
+ DAMON_ARBITRARY_TARGET,
+};
+
/**
* struct damon_ctx - Represents a context for each monitoring. This is the
* main interface that allows users to set the attributes and get the results
@@ -130,7 +185,15 @@ struct damon_callback {
* @primitive: Set of monitoring primitives for given use cases.
* @callback: Set of callbacks for monitoring events notifications.
*
- * @target: Pointer to the user-defined monitoring target.
+ * @target_type: Type of the monitoring target.
+ *
+ * @region_targets: Head of monitoring targets (&damon_target) list.
+ *
+ * @arbitrary_target: Pointer to arbitrary type target.
+ *
+ * @region_targets are valid only if @target_type is
+ * DAMON_REGION_SAMPLING_TARGET. @arbitrary_target is valid only if
+ * @target_type is DAMON_ARBITRARY_TARGET.
*/
struct damon_ctx {
unsigned long sample_interval;
@@ -149,12 +212,48 @@ struct damon_ctx {
struct damon_primitive primitive;
struct damon_callback callback;
- void *target;
+ enum damon_target_type target_type;
+ union {
+ /* DAMON_REGION_SAMPLING_TARGET */
+ struct list_head region_targets;
+
+ /* DAMON_ARBITRARY_TARGET */
+ void *arbitrary_target;
+ };
};
+#define damon_next_region(r) \
+ (container_of(r->list.next, struct damon_region, list))
+
+#define damon_prev_region(r) \
+ (container_of(r->list.prev, struct damon_region, list))
+
+#define damon_for_each_region(r, t) \
+ list_for_each_entry(r, &t->regions_list, list)
+
+#define damon_for_each_region_safe(r, next, t) \
+ list_for_each_entry_safe(r, next, &t->regions_list, list)
+
+#define damon_for_each_target(t, ctx) \
+ list_for_each_entry(t, &(ctx)->region_targets, list)
+
+#define damon_for_each_target_safe(t, next, ctx) \
+ list_for_each_entry_safe(t, next, &(ctx)->region_targets, list)
+
#ifdef CONFIG_DAMON
-struct damon_ctx *damon_new_ctx(void);
+struct damon_region *damon_new_region(unsigned long start, unsigned long end);
+inline void damon_insert_region(struct damon_region *r,
+ struct damon_region *prev, struct damon_region *next);
+void damon_add_region(struct damon_region *r, struct damon_target *t);
+void damon_destroy_region(struct damon_region *r);
+
+struct damon_target *damon_new_target(unsigned long id);
+void damon_add_target(struct damon_ctx *ctx, struct damon_target *t);
+void damon_free_target(struct damon_target *t);
+void damon_destroy_target(struct damon_target *t);
+
+struct damon_ctx *damon_new_ctx(enum damon_target_type type);
void damon_destroy_ctx(struct damon_ctx *ctx);
int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
unsigned long aggr_int, unsigned long regions_update_int);
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 8963804efdf9..167487e75737 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -15,7 +15,102 @@
static DEFINE_MUTEX(damon_lock);
static int nr_running_ctxs;
-struct damon_ctx *damon_new_ctx(void)
+/*
+ * Construct a damon_region struct
+ *
+ * Returns the pointer to the new struct if success, or NULL otherwise
+ */
+struct damon_region *damon_new_region(unsigned long start, unsigned long end)
+{
+ struct damon_region *region;
+
+ region = kmalloc(sizeof(*region), GFP_KERNEL);
+ if (!region)
+ return NULL;
+
+ region->ar.start = start;
+ region->ar.end = end;
+ region->nr_accesses = 0;
+ INIT_LIST_HEAD(®ion->list);
+
+ return region;
+}
+
+/*
+ * Add a region between two other regions
+ */
+inline void damon_insert_region(struct damon_region *r,
+ struct damon_region *prev, struct damon_region *next)
+{
+ __list_add(&r->list, &prev->list, &next->list);
+}
+
+void damon_add_region(struct damon_region *r, struct damon_target *t)
+{
+ list_add_tail(&r->list, &t->regions_list);
+}
+
+static void damon_del_region(struct damon_region *r)
+{
+ list_del(&r->list);
+}
+
+static void damon_free_region(struct damon_region *r)
+{
+ kfree(r);
+}
+
+void damon_destroy_region(struct damon_region *r)
+{
+ damon_del_region(r);
+ damon_free_region(r);
+}
+
+/*
+ * Construct a damon_target struct
+ *
+ * Returns the pointer to the new struct if success, or NULL otherwise
+ */
+struct damon_target *damon_new_target(unsigned long id)
+{
+ struct damon_target *t;
+
+ t = kmalloc(sizeof(*t), GFP_KERNEL);
+ if (!t)
+ return NULL;
+
+ t->id = id;
+ INIT_LIST_HEAD(&t->regions_list);
+
+ return t;
+}
+
+void damon_add_target(struct damon_ctx *ctx, struct damon_target *t)
+{
+ list_add_tail(&t->list, &ctx->region_targets);
+}
+
+static void damon_del_target(struct damon_target *t)
+{
+ list_del(&t->list);
+}
+
+void damon_free_target(struct damon_target *t)
+{
+ struct damon_region *r, *next;
+
+ damon_for_each_region_safe(r, next, t)
+ damon_free_region(r);
+ kfree(t);
+}
+
+void damon_destroy_target(struct damon_target *t)
+{
+ damon_del_target(t);
+ damon_free_target(t);
+}
+
+struct damon_ctx *damon_new_ctx(enum damon_target_type type)
{
struct damon_ctx *ctx;
@@ -32,13 +127,20 @@ struct damon_ctx *damon_new_ctx(void)
mutex_init(&ctx->kdamond_lock);
- ctx->target = NULL;
+ ctx->target_type = type;
+ if (type != DAMON_ARBITRARY_TARGET)
+ INIT_LIST_HEAD(&ctx->region_targets);
return ctx;
}
void damon_destroy_ctx(struct damon_ctx *ctx)
{
+ struct damon_target *t, *next_t;
+
+ damon_for_each_target_safe(t, next_t, ctx)
+ damon_destroy_target(t);
+
kfree(ctx);
}
@@ -215,6 +317,21 @@ static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx)
ctx->aggr_interval);
}
+/*
+ * Reset the aggregated monitoring results ('nr_accesses' of each region).
+ */
+static void kdamond_reset_aggregated(struct damon_ctx *c)
+{
+ struct damon_target *t;
+
+ damon_for_each_target(t, c) {
+ struct damon_region *r;
+
+ damon_for_each_region(r, t)
+ r->nr_accesses = 0;
+ }
+}
+
/*
* Check whether it is time to check and apply the target monitoring regions
*
@@ -236,6 +353,7 @@ static bool kdamond_need_update_regions(struct damon_ctx *ctx)
*/
static bool kdamond_need_stop(struct damon_ctx *ctx)
{
+ struct damon_target *t;
bool stop;
mutex_lock(&ctx->kdamond_lock);
@@ -247,7 +365,15 @@ static bool kdamond_need_stop(struct damon_ctx *ctx)
if (!ctx->primitive.target_valid)
return false;
- return !ctx->primitive.target_valid(ctx->target);
+ if (ctx->target_type == DAMON_ARBITRARY_TARGET)
+ return !ctx->primitive.target_valid(ctx->arbitrary_target);
+
+ damon_for_each_target(t, ctx) {
+ if (ctx->primitive.target_valid(t))
+ return false;
+ }
+
+ return true;
}
static void set_kdamond_stop(struct damon_ctx *ctx)
@@ -263,6 +389,8 @@ static void set_kdamond_stop(struct damon_ctx *ctx)
static int kdamond_fn(void *data)
{
struct damon_ctx *ctx = (struct damon_ctx *)data;
+ struct damon_target *t;
+ struct damon_region *r, *next;
pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
@@ -287,6 +415,8 @@ static int kdamond_fn(void *data)
if (ctx->callback.after_aggregation &&
ctx->callback.after_aggregation(ctx))
set_kdamond_stop(ctx);
+ if (ctx->target_type != DAMON_ARBITRARY_TARGET)
+ kdamond_reset_aggregated(ctx);
if (ctx->primitive.reset_aggregated)
ctx->primitive.reset_aggregated(ctx);
}
@@ -296,6 +426,12 @@ static int kdamond_fn(void *data)
ctx->primitive.update_target_regions(ctx);
}
}
+ if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
+ damon_for_each_target(t, ctx) {
+ damon_for_each_region_safe(r, next, t)
+ damon_destroy_region(r);
+ }
+ }
if (ctx->callback.before_terminate &&
ctx->callback.before_terminate(ctx))
--
2.17.1
From: SeongJae Park <[email protected]>
Even somehow the initial monitoring target regions are well constructed
to fulfill the assumption (pages in same region have similar access
frequencies), the data access pattern can be dynamically changed. This
will result in low monitoring quality. To keep the assumption as much
as possible, DAMON adaptively merges and splits each region based on
their access frequency.
For each ``aggregation interval``, it compares the access frequencies of
adjacent regions and merges those if the frequency difference is small.
Then, after it reports and clears the aggregated access frequency of
each region, it splits each region into two or three regions if the
total number of regions will not exceed the user-specified maximum
number of regions after the split.
In this way, DAMON provides its best-effort quality and minimal overhead
while keeping the upper-bound overhead that users set.
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Leonard Foerster <[email protected]>
---
include/linux/damon.h | 41 +++++---
mm/damon/core.c | 220 ++++++++++++++++++++++++++++++++++++++++--
2 files changed, 240 insertions(+), 21 deletions(-)
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 7d4685adc8a9..f446f8433599 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -12,6 +12,9 @@
#include <linux/time64.h>
#include <linux/types.h>
+/* Minimal region size. Every damon_region is aligned by this. */
+#define DAMON_MIN_REGION PAGE_SIZE
+
/**
* struct damon_addr_range - Represents an address region of [@start, @end).
* @start: Start address of the region (inclusive).
@@ -86,6 +89,8 @@ struct damon_ctx;
* prepared for the next access check.
* @check_accesses should check the accesses to each region that made after the
* last preparation and update the number of observed accesses of each region.
+ * It should also return max number of observed accesses that made as a result
+ * of its update.
* @reset_aggregated should reset the access monitoring results that aggregated
* by @check_accesses.
* @target_valid should check whether the target is still valid for the
@@ -98,7 +103,7 @@ struct damon_primitive {
void (*init_target_regions)(struct damon_ctx *context);
void (*update_target_regions)(struct damon_ctx *context);
void (*prepare_access_checks)(struct damon_ctx *context);
- void (*check_accesses)(struct damon_ctx *context);
+ unsigned int (*check_accesses)(struct damon_ctx *context);
void (*reset_aggregated)(struct damon_ctx *context);
bool (*target_valid)(void *target);
void (*cleanup)(struct damon_ctx *context);
@@ -138,11 +143,11 @@ struct damon_callback {
/**
* enum damon_target_type - Represents the type of the monitoring target.
*
- * @DAMON_REGION_SAMPLING_TARGET: Region based sampling target.
- * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
+ * @DAMON_ADAPTIVE_TARGET: Adaptive regions adjustment applied target.
+ * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
*/
enum damon_target_type {
- DAMON_REGION_SAMPLING_TARGET,
+ DAMON_ADAPTIVE_TARGET,
DAMON_ARBITRARY_TARGET,
};
@@ -187,13 +192,15 @@ enum damon_target_type {
*
* @target_type: Type of the monitoring target.
*
- * @region_targets: Head of monitoring targets (&damon_target) list.
+ * @min_nr_regions: The minimum number of adaptive monitoring regions.
+ * @max_nr_regions: The maximum number of adaptive monitoring regions.
+ * @adaptive_targets: Head of monitoring targets (&damon_target) list.
*
* @arbitrary_target: Pointer to arbitrary type target.
*
- * @region_targets are valid only if @target_type is
- * DAMON_REGION_SAMPLING_TARGET. @arbitrary_target is valid only if
- * @target_type is DAMON_ARBITRARY_TARGET.
+ * @min_nr_regions, @max_nr_regions and @adaptive_targets are valid only if
+ * @target_type is &DAMON_ADAPTIVE_TARGET. @arbitrary_target is valid only if
+ * @target_type is &DAMON_ARBITRARY_TARGET.
*/
struct damon_ctx {
unsigned long sample_interval;
@@ -214,11 +221,13 @@ struct damon_ctx {
enum damon_target_type target_type;
union {
- /* DAMON_REGION_SAMPLING_TARGET */
- struct list_head region_targets;
+ struct { /* DAMON_ADAPTIVE_TARGET */
+ unsigned long min_nr_regions;
+ unsigned long max_nr_regions;
+ struct list_head adaptive_targets;
+ };
- /* DAMON_ARBITRARY_TARGET */
- void *arbitrary_target;
+ void *arbitrary_target; /* DAMON_ARBITRARY_TARGET */
};
};
@@ -235,10 +244,10 @@ struct damon_ctx {
list_for_each_entry_safe(r, next, &t->regions_list, list)
#define damon_for_each_target(t, ctx) \
- list_for_each_entry(t, &(ctx)->region_targets, list)
+ list_for_each_entry(t, &(ctx)->adaptive_targets, list)
#define damon_for_each_target_safe(t, next, ctx) \
- list_for_each_entry_safe(t, next, &(ctx)->region_targets, list)
+ list_for_each_entry_safe(t, next, &(ctx)->adaptive_targets, list)
#ifdef CONFIG_DAMON
@@ -252,11 +261,13 @@ struct damon_target *damon_new_target(unsigned long id);
void damon_add_target(struct damon_ctx *ctx, struct damon_target *t);
void damon_free_target(struct damon_target *t);
void damon_destroy_target(struct damon_target *t);
+unsigned int damon_nr_regions(struct damon_target *t);
struct damon_ctx *damon_new_ctx(enum damon_target_type type);
void damon_destroy_ctx(struct damon_ctx *ctx);
int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
- unsigned long aggr_int, unsigned long regions_update_int);
+ unsigned long aggr_int, unsigned long regions_update_int,
+ unsigned long min_nr_reg, unsigned long max_nr_reg);
int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 167487e75737..0f9beb60d9dd 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -10,8 +10,12 @@
#include <linux/damon.h>
#include <linux/delay.h>
#include <linux/kthread.h>
+#include <linux/random.h>
#include <linux/slab.h>
+/* Get a random number in [l, r) */
+#define damon_rand(l, r) (l + prandom_u32_max(r - l))
+
static DEFINE_MUTEX(damon_lock);
static int nr_running_ctxs;
@@ -87,7 +91,7 @@ struct damon_target *damon_new_target(unsigned long id)
void damon_add_target(struct damon_ctx *ctx, struct damon_target *t)
{
- list_add_tail(&t->list, &ctx->region_targets);
+ list_add_tail(&t->list, &ctx->adaptive_targets);
}
static void damon_del_target(struct damon_target *t)
@@ -110,6 +114,17 @@ void damon_destroy_target(struct damon_target *t)
damon_free_target(t);
}
+unsigned int damon_nr_regions(struct damon_target *t)
+{
+ struct damon_region *r;
+ unsigned int nr_regions = 0;
+
+ damon_for_each_region(r, t)
+ nr_regions++;
+
+ return nr_regions;
+}
+
struct damon_ctx *damon_new_ctx(enum damon_target_type type)
{
struct damon_ctx *ctx;
@@ -128,8 +143,12 @@ struct damon_ctx *damon_new_ctx(enum damon_target_type type)
mutex_init(&ctx->kdamond_lock);
ctx->target_type = type;
- if (type != DAMON_ARBITRARY_TARGET)
- INIT_LIST_HEAD(&ctx->region_targets);
+ if (type != DAMON_ARBITRARY_TARGET) {
+ ctx->min_nr_regions = 10;
+ ctx->max_nr_regions = 1000;
+
+ INIT_LIST_HEAD(&ctx->adaptive_targets);
+ }
return ctx;
}
@@ -150,6 +169,8 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
* @sample_int: time interval between samplings
* @aggr_int: time interval between aggregations
* @regions_update_int: time interval between target regions update
+ * @min_nr_reg: minimal number of regions
+ * @max_nr_reg: maximum number of regions
*
* This function should not be called while the kdamond is running.
* Every time interval is in micro-seconds.
@@ -157,15 +178,51 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
* Return: 0 on success, negative error code otherwise.
*/
int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
- unsigned long aggr_int, unsigned long regions_update_int)
+ unsigned long aggr_int, unsigned long regions_update_int,
+ unsigned long min_nr_reg, unsigned long max_nr_reg)
{
+ if (min_nr_reg < 3) {
+ pr_err("min_nr_regions (%lu) must be at least 3\n",
+ min_nr_reg);
+ return -EINVAL;
+ }
+ if (min_nr_reg > max_nr_reg) {
+ pr_err("invalid nr_regions. min (%lu) > max (%lu)\n",
+ min_nr_reg, max_nr_reg);
+ return -EINVAL;
+ }
+
ctx->sample_interval = sample_int;
ctx->aggr_interval = aggr_int;
ctx->regions_update_interval = regions_update_int;
+ if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
+ ctx->min_nr_regions = min_nr_reg;
+ ctx->max_nr_regions = max_nr_reg;
+ }
return 0;
}
+/* Returns the size upper limit for each monitoring region */
+static unsigned long damon_region_sz_limit(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct damon_region *r;
+ unsigned long sz = 0;
+
+ damon_for_each_target(t, ctx) {
+ damon_for_each_region(r, t)
+ sz += r->ar.end - r->ar.start;
+ }
+
+ if (ctx->min_nr_regions)
+ sz /= ctx->min_nr_regions;
+ if (sz < DAMON_MIN_REGION)
+ sz = DAMON_MIN_REGION;
+
+ return sz;
+}
+
static bool damon_kdamond_running(struct damon_ctx *ctx)
{
bool running;
@@ -332,6 +389,146 @@ static void kdamond_reset_aggregated(struct damon_ctx *c)
}
}
+#define sz_damon_region(r) (r->ar.end - r->ar.start)
+
+/*
+ * Merge two adjacent regions into one region
+ */
+static void damon_merge_two_regions(struct damon_region *l,
+ struct damon_region *r)
+{
+ unsigned long sz_l = sz_damon_region(l), sz_r = sz_damon_region(r);
+
+ l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
+ (sz_l + sz_r);
+ l->ar.end = r->ar.end;
+ damon_destroy_region(r);
+}
+
+#define diff_of(a, b) (a > b ? a - b : b - a)
+
+/*
+ * Merge adjacent regions having similar access frequencies
+ *
+ * t target affected by this merge operation
+ * thres '->nr_accesses' diff threshold for the merge
+ * sz_limit size upper limit of each region
+ */
+static void damon_merge_regions_of(struct damon_target *t, unsigned int thres,
+ unsigned long sz_limit)
+{
+ struct damon_region *r, *prev = NULL, *next;
+
+ damon_for_each_region_safe(r, next, t) {
+ if (prev && prev->ar.end == r->ar.start &&
+ diff_of(prev->nr_accesses, r->nr_accesses) <= thres &&
+ sz_damon_region(prev) + sz_damon_region(r) <= sz_limit)
+ damon_merge_two_regions(prev, r);
+ else
+ prev = r;
+ }
+}
+
+/*
+ * Merge adjacent regions having similar access frequencies
+ *
+ * threshold '->nr_accesses' diff threshold for the merge
+ * sz_limit size upper limit of each region
+ *
+ * This function merges monitoring target regions which are adjacent and their
+ * access frequencies are similar. This is for minimizing the monitoring
+ * overhead under the dynamically changeable access pattern. If a merge was
+ * unnecessarily made, later 'kdamond_split_regions()' will revert it.
+ */
+static void kdamond_merge_regions(struct damon_ctx *c, unsigned int threshold,
+ unsigned long sz_limit)
+{
+ struct damon_target *t;
+
+ damon_for_each_target(t, c)
+ damon_merge_regions_of(t, threshold, sz_limit);
+}
+
+/*
+ * Split a region in two
+ *
+ * r the region to be split
+ * sz_r size of the first sub-region that will be made
+ */
+static void damon_split_region_at(struct damon_ctx *ctx,
+ struct damon_region *r, unsigned long sz_r)
+{
+ struct damon_region *new;
+
+ new = damon_new_region(r->ar.start + sz_r, r->ar.end);
+ r->ar.end = new->ar.start;
+
+ damon_insert_region(new, r, damon_next_region(r));
+}
+
+/* Split every region in the given target into 'nr_subs' regions */
+static void damon_split_regions_of(struct damon_ctx *ctx,
+ struct damon_target *t, int nr_subs)
+{
+ struct damon_region *r, *next;
+ unsigned long sz_region, sz_sub = 0;
+ int i;
+
+ damon_for_each_region_safe(r, next, t) {
+ sz_region = r->ar.end - r->ar.start;
+
+ for (i = 0; i < nr_subs - 1 &&
+ sz_region > 2 * DAMON_MIN_REGION; i++) {
+ /*
+ * Randomly select size of left sub-region to be at
+ * least 10 percent and at most 90% of original region
+ */
+ sz_sub = ALIGN_DOWN(damon_rand(1, 10) *
+ sz_region / 10, DAMON_MIN_REGION);
+ /* Do not allow blank region */
+ if (sz_sub == 0 || sz_sub >= sz_region)
+ continue;
+
+ damon_split_region_at(ctx, r, sz_sub);
+ sz_region = sz_sub;
+ }
+ }
+}
+
+/*
+ * Split every target region into randomly-sized small regions
+ *
+ * This function splits every target region into random-sized small regions if
+ * current total number of the regions is equal or smaller than half of the
+ * user-specified maximum number of regions. This is for maximizing the
+ * monitoring accuracy under the dynamically changeable access patterns. If a
+ * split was unnecessarily made, later 'kdamond_merge_regions()' will revert
+ * it.
+ */
+static void kdamond_split_regions(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ unsigned int nr_regions = 0;
+ static unsigned int last_nr_regions;
+ int nr_subregions = 2;
+
+ damon_for_each_target(t, ctx)
+ nr_regions += damon_nr_regions(t);
+
+ if (nr_regions > ctx->max_nr_regions / 2)
+ return;
+
+ /* Maybe the middle of the region has different access frequency */
+ if (last_nr_regions == nr_regions &&
+ nr_regions < ctx->max_nr_regions / 3)
+ nr_subregions = 3;
+
+ damon_for_each_target(t, ctx)
+ damon_split_regions_of(ctx, t, nr_subregions);
+
+ last_nr_regions = nr_regions;
+}
+
/*
* Check whether it is time to check and apply the target monitoring regions
*
@@ -391,6 +588,8 @@ static int kdamond_fn(void *data)
struct damon_ctx *ctx = (struct damon_ctx *)data;
struct damon_target *t;
struct damon_region *r, *next;
+ unsigned int max_nr_accesses = 0;
+ unsigned long sz_limit = 0;
pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
@@ -399,6 +598,8 @@ static int kdamond_fn(void *data)
if (ctx->callback.before_start && ctx->callback.before_start(ctx))
set_kdamond_stop(ctx);
+ sz_limit = damon_region_sz_limit(ctx);
+
while (!kdamond_need_stop(ctx)) {
if (ctx->primitive.prepare_access_checks)
ctx->primitive.prepare_access_checks(ctx);
@@ -409,14 +610,20 @@ static int kdamond_fn(void *data)
usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
if (ctx->primitive.check_accesses)
- ctx->primitive.check_accesses(ctx);
+ max_nr_accesses = ctx->primitive.check_accesses(ctx);
if (kdamond_aggregate_interval_passed(ctx)) {
+ if (ctx->target_type != DAMON_ARBITRARY_TARGET)
+ kdamond_merge_regions(ctx,
+ max_nr_accesses / 10,
+ sz_limit);
if (ctx->callback.after_aggregation &&
ctx->callback.after_aggregation(ctx))
set_kdamond_stop(ctx);
- if (ctx->target_type != DAMON_ARBITRARY_TARGET)
+ if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
kdamond_reset_aggregated(ctx);
+ kdamond_split_regions(ctx);
+ }
if (ctx->primitive.reset_aggregated)
ctx->primitive.reset_aggregated(ctx);
}
@@ -424,6 +631,7 @@ static int kdamond_fn(void *data)
if (kdamond_need_update_regions(ctx)) {
if (ctx->primitive.update_target_regions)
ctx->primitive.update_target_regions(ctx);
+ sz_limit = damon_region_sz_limit(ctx);
}
}
if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
--
2.17.1
From: SeongJae Park <[email protected]>
This commit introduces a reference implementation of the address space
specific low level primitives for the virtual address space, so that
users of DAMON can easily monitor the data accesses on virtual address
spaces of specific processes by simply configuring the implementation to
be used by DAMON.
The low level primitives for the fundamental access monitoring are
defined in two parts:
1. Identification of the monitoring target address range for the address
space.
2. Access check of specific address range in the target space.
The reference implementation for the virtual address space does the
works as below.
PTE Accessed-bit Based Access Check
-----------------------------------
The implementation uses PTE Accessed-bit for basic access checks. That
is, it clears the bit for next sampling target page and checks whether
it set again after one sampling period. This could disturb the reclaim
logic. DAMON uses ``PG_idle`` and ``PG_young`` page flags to solve the
conflict, as Idle page tracking does.
VMA-based Target Address Range Construction
-------------------------------------------
Only small parts in the super-huge virtual address space of the
processes are mapped to physical memory and accessed. Thus, tracking
the unmapped address regions is just wasteful. However, because DAMON
can deal with some level of noise using the adaptive regions adjustment
mechanism, tracking every mapping is not strictly required but could
even incur a high overhead in some cases. That said, too huge unmapped
areas inside the monitoring target should be removed to not take the
time for the adaptive mechanism.
For the reason, this implementation converts the complex mappings to
three distinct regions that cover every mapped area of the address
space. Also, the two gaps between the three regions are the two biggest
unmapped areas in the given address space. The two biggest unmapped
areas would be the gap between the heap and the uppermost mmap()-ed
region, and the gap between the lowermost mmap()-ed region and the stack
in most of the cases. Because these gaps are exceptionally huge in
usual address spacees, excluding these will be sufficient to make a
reasonable trade-off. Below shows this in detail::
<heap>
<BIG UNMAPPED REGION 1>
<uppermost mmap()-ed region>
(small mmap()-ed regions and munmap()-ed regions)
<lowermost mmap()-ed region>
<BIG UNMAPPED REGION 2>
<stack>
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Leonard Foerster <[email protected]>
---
include/linux/damon.h | 13 +
mm/damon/Kconfig | 9 +
mm/damon/Makefile | 1 +
mm/damon/vaddr.c | 579 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 602 insertions(+)
create mode 100644 mm/damon/vaddr.c
diff --git a/include/linux/damon.h b/include/linux/damon.h
index f446f8433599..39b4d6d3ddee 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -274,4 +274,17 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
#endif /* CONFIG_DAMON */
+#ifdef CONFIG_DAMON_VADDR
+
+/* Monitoring primitives for virtual memory address spaces */
+void damon_va_init_regions(struct damon_ctx *ctx);
+void damon_va_update_regions(struct damon_ctx *ctx);
+void damon_va_prepare_access_checks(struct damon_ctx *ctx);
+unsigned int damon_va_check_accesses(struct damon_ctx *ctx);
+bool damon_va_target_valid(void *t);
+void damon_va_cleanup(struct damon_ctx *ctx);
+void damon_va_set_primitives(struct damon_ctx *ctx);
+
+#endif /* CONFIG_DAMON_VADDR */
+
#endif /* _DAMON_H */
diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
index d00e99ac1a15..8ae080c52950 100644
--- a/mm/damon/Kconfig
+++ b/mm/damon/Kconfig
@@ -12,4 +12,13 @@ config DAMON
See https://damonitor.github.io/doc/html/latest-damon/index.html for
more information.
+config DAMON_VADDR
+ bool "Data access monitoring primitives for virtual address spaces"
+ depends on DAMON && MMU
+ select PAGE_EXTENSION if !64BIT
+ select PAGE_IDLE_FLAG
+ help
+ This builds the default data access monitoring primitives for DAMON
+ that works for virtual address spaces.
+
endmenu
diff --git a/mm/damon/Makefile b/mm/damon/Makefile
index 4fd2edb4becf..6ebbd08aed67 100644
--- a/mm/damon/Makefile
+++ b/mm/damon/Makefile
@@ -1,3 +1,4 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_DAMON) := core.o
+obj-$(CONFIG_DAMON_VADDR) += vaddr.o
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
new file mode 100644
index 000000000000..efc56ef843fe
--- /dev/null
+++ b/mm/damon/vaddr.c
@@ -0,0 +1,579 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DAMON Primitives for Virtual Address Spaces
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#define pr_fmt(fmt) "damon-va: " fmt
+
+#include <linux/damon.h>
+#include <linux/mm.h>
+#include <linux/mmu_notifier.h>
+#include <linux/page_idle.h>
+#include <linux/random.h>
+#include <linux/sched/mm.h>
+#include <linux/slab.h>
+
+/* Get a random number in [l, r) */
+#define damon_rand(l, r) (l + prandom_u32_max(r - l))
+
+/*
+ * 't->id' should be the pointer to the relevant 'struct pid' having reference
+ * count. Caller must put the returned task, unless it is NULL.
+ */
+#define damon_get_task_struct(t) \
+ (get_pid_task((struct pid *)t->id, PIDTYPE_PID))
+
+/*
+ * Get the mm_struct of the given target
+ *
+ * Caller _must_ put the mm_struct after use, unless it is NULL.
+ *
+ * Returns the mm_struct of the target on success, NULL on failure
+ */
+static struct mm_struct *damon_get_mm(struct damon_target *t)
+{
+ struct task_struct *task;
+ struct mm_struct *mm;
+
+ task = damon_get_task_struct(t);
+ if (!task)
+ return NULL;
+
+ mm = get_task_mm(task);
+ put_task_struct(task);
+ return mm;
+}
+
+/*
+ * Functions for the initial monitoring target regions construction
+ */
+
+/*
+ * Size-evenly split a region into 'nr_pieces' small regions
+ *
+ * Returns 0 on success, or negative error code otherwise.
+ */
+static int damon_va_evenly_split_region(struct damon_ctx *ctx,
+ struct damon_region *r, unsigned int nr_pieces)
+{
+ unsigned long sz_orig, sz_piece, orig_end;
+ struct damon_region *n = NULL, *next;
+ unsigned long start;
+
+ if (!r || !nr_pieces)
+ return -EINVAL;
+
+ orig_end = r->ar.end;
+ sz_orig = r->ar.end - r->ar.start;
+ sz_piece = ALIGN_DOWN(sz_orig / nr_pieces, DAMON_MIN_REGION);
+
+ if (!sz_piece)
+ return -EINVAL;
+
+ r->ar.end = r->ar.start + sz_piece;
+ next = damon_next_region(r);
+ for (start = r->ar.end; start + sz_piece <= orig_end;
+ start += sz_piece) {
+ n = damon_new_region(start, start + sz_piece);
+ if (!n)
+ return -ENOMEM;
+ damon_insert_region(n, r, next);
+ r = n;
+ }
+ /* complement last region for possible rounding error */
+ if (n)
+ n->ar.end = orig_end;
+
+ return 0;
+}
+
+static unsigned long sz_range(struct damon_addr_range *r)
+{
+ return r->end - r->start;
+}
+
+static void swap_ranges(struct damon_addr_range *r1,
+ struct damon_addr_range *r2)
+{
+ struct damon_addr_range tmp;
+
+ tmp = *r1;
+ *r1 = *r2;
+ *r2 = tmp;
+}
+
+/*
+ * Find three regions separated by two biggest unmapped regions
+ *
+ * vma the head vma of the target address space
+ * regions an array of three address ranges that results will be saved
+ *
+ * This function receives an address space and finds three regions in it which
+ * separated by the two biggest unmapped regions in the space. Please refer to
+ * below comments of '__damon_va_init_regions()' function to know why this is
+ * necessary.
+ *
+ * Returns 0 if success, or negative error code otherwise.
+ */
+static int __damon_va_three_regions(struct vm_area_struct *vma,
+ struct damon_addr_range regions[3])
+{
+ struct damon_addr_range gap = {0}, first_gap = {0}, second_gap = {0};
+ struct vm_area_struct *last_vma = NULL;
+ unsigned long start = 0;
+ struct rb_root rbroot;
+
+ /* Find two biggest gaps so that first_gap > second_gap > others */
+ for (; vma; vma = vma->vm_next) {
+ if (!last_vma) {
+ start = vma->vm_start;
+ goto next;
+ }
+
+ if (vma->rb_subtree_gap <= sz_range(&second_gap)) {
+ rbroot.rb_node = &vma->vm_rb;
+ vma = rb_entry(rb_last(&rbroot),
+ struct vm_area_struct, vm_rb);
+ goto next;
+ }
+
+ gap.start = last_vma->vm_end;
+ gap.end = vma->vm_start;
+ if (sz_range(&gap) > sz_range(&second_gap)) {
+ swap_ranges(&gap, &second_gap);
+ if (sz_range(&second_gap) > sz_range(&first_gap))
+ swap_ranges(&second_gap, &first_gap);
+ }
+next:
+ last_vma = vma;
+ }
+
+ if (!sz_range(&second_gap) || !sz_range(&first_gap))
+ return -EINVAL;
+
+ /* Sort the two biggest gaps by address */
+ if (first_gap.start > second_gap.start)
+ swap_ranges(&first_gap, &second_gap);
+
+ /* Store the result */
+ regions[0].start = ALIGN(start, DAMON_MIN_REGION);
+ regions[0].end = ALIGN(first_gap.start, DAMON_MIN_REGION);
+ regions[1].start = ALIGN(first_gap.end, DAMON_MIN_REGION);
+ regions[1].end = ALIGN(second_gap.start, DAMON_MIN_REGION);
+ regions[2].start = ALIGN(second_gap.end, DAMON_MIN_REGION);
+ regions[2].end = ALIGN(last_vma->vm_end, DAMON_MIN_REGION);
+
+ return 0;
+}
+
+/*
+ * Get the three regions in the given target (task)
+ *
+ * Returns 0 on success, negative error code otherwise.
+ */
+static int damon_va_three_regions(struct damon_target *t,
+ struct damon_addr_range regions[3])
+{
+ struct mm_struct *mm;
+ int rc;
+
+ mm = damon_get_mm(t);
+ if (!mm)
+ return -EINVAL;
+
+ mmap_read_lock(mm);
+ rc = __damon_va_three_regions(mm->mmap, regions);
+ mmap_read_unlock(mm);
+
+ mmput(mm);
+ return rc;
+}
+
+/*
+ * Initialize the monitoring target regions for the given target (task)
+ *
+ * t the given target
+ *
+ * Because only a number of small portions of the entire address space
+ * is actually mapped to the memory and accessed, monitoring the unmapped
+ * regions is wasteful. That said, because we can deal with small noises,
+ * tracking every mapping is not strictly required but could even incur a high
+ * overhead if the mapping frequently changes or the number of mappings is
+ * high. The adaptive regions adjustment mechanism will further help to deal
+ * with the noise by simply identifying the unmapped areas as a region that
+ * has no access. Moreover, applying the real mappings that would have many
+ * unmapped areas inside will make the adaptive mechanism quite complex. That
+ * said, too huge unmapped areas inside the monitoring target should be removed
+ * to not take the time for the adaptive mechanism.
+ *
+ * For the reason, we convert the complex mappings to three distinct regions
+ * that cover every mapped area of the address space. Also the two gaps
+ * between the three regions are the two biggest unmapped areas in the given
+ * address space. In detail, this function first identifies the start and the
+ * end of the mappings and the two biggest unmapped areas of the address space.
+ * Then, it constructs the three regions as below:
+ *
+ * [mappings[0]->start, big_two_unmapped_areas[0]->start)
+ * [big_two_unmapped_areas[0]->end, big_two_unmapped_areas[1]->start)
+ * [big_two_unmapped_areas[1]->end, mappings[nr_mappings - 1]->end)
+ *
+ * As usual memory map of processes is as below, the gap between the heap and
+ * the uppermost mmap()-ed region, and the gap between the lowermost mmap()-ed
+ * region and the stack will be two biggest unmapped regions. Because these
+ * gaps are exceptionally huge areas in usual address space, excluding these
+ * two biggest unmapped regions will be sufficient to make a trade-off.
+ *
+ * <heap>
+ * <BIG UNMAPPED REGION 1>
+ * <uppermost mmap()-ed region>
+ * (other mmap()-ed regions and small unmapped regions)
+ * <lowermost mmap()-ed region>
+ * <BIG UNMAPPED REGION 2>
+ * <stack>
+ */
+static void __damon_va_init_regions(struct damon_ctx *c,
+ struct damon_target *t)
+{
+ struct damon_region *r;
+ struct damon_addr_range regions[3];
+ unsigned long sz = 0, nr_pieces;
+ int i;
+
+ if (damon_va_three_regions(t, regions)) {
+ pr_err("Failed to get three regions of target %lu\n", t->id);
+ return;
+ }
+
+ for (i = 0; i < 3; i++)
+ sz += regions[i].end - regions[i].start;
+ if (c->min_nr_regions)
+ sz /= c->min_nr_regions;
+ if (sz < DAMON_MIN_REGION)
+ sz = DAMON_MIN_REGION;
+
+ /* Set the initial three regions of the target */
+ for (i = 0; i < 3; i++) {
+ r = damon_new_region(regions[i].start, regions[i].end);
+ if (!r) {
+ pr_err("%d'th init region creation failed\n", i);
+ return;
+ }
+ damon_add_region(r, t);
+
+ nr_pieces = (regions[i].end - regions[i].start) / sz;
+ damon_va_evenly_split_region(c, r, nr_pieces);
+ }
+}
+
+/* Initialize '->regions_list' of every target (task) */
+void damon_va_init_regions(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+
+ damon_for_each_target(t, ctx) {
+ /* the user may set the target regions as they want */
+ if (!damon_nr_regions(t))
+ __damon_va_init_regions(ctx, t);
+ }
+}
+
+/*
+ * Functions for the dynamic monitoring target regions update
+ */
+
+/*
+ * Check whether a region is intersecting an address range
+ *
+ * Returns true if it is.
+ */
+static bool damon_intersect(struct damon_region *r, struct damon_addr_range *re)
+{
+ return !(r->ar.end <= re->start || re->end <= r->ar.start);
+}
+
+/*
+ * Update damon regions for the three big regions of the given target
+ *
+ * t the given target
+ * bregions the three big regions of the target
+ */
+static void damon_va_apply_three_regions(struct damon_ctx *ctx,
+ struct damon_target *t, struct damon_addr_range bregions[3])
+{
+ struct damon_region *r, *next;
+ unsigned int i = 0;
+
+ /* Remove regions which are not in the three big regions now */
+ damon_for_each_region_safe(r, next, t) {
+ for (i = 0; i < 3; i++) {
+ if (damon_intersect(r, &bregions[i]))
+ break;
+ }
+ if (i == 3)
+ damon_destroy_region(r);
+ }
+
+ /* Adjust intersecting regions to fit with the three big regions */
+ for (i = 0; i < 3; i++) {
+ struct damon_region *first = NULL, *last;
+ struct damon_region *newr;
+ struct damon_addr_range *br;
+
+ br = &bregions[i];
+ /* Get the first and last regions which intersects with br */
+ damon_for_each_region(r, t) {
+ if (damon_intersect(r, br)) {
+ if (!first)
+ first = r;
+ last = r;
+ }
+ if (r->ar.start >= br->end)
+ break;
+ }
+ if (!first) {
+ /* no damon_region intersects with this big region */
+ newr = damon_new_region(
+ ALIGN_DOWN(br->start,
+ DAMON_MIN_REGION),
+ ALIGN(br->end, DAMON_MIN_REGION));
+ if (!newr)
+ continue;
+ damon_insert_region(newr, damon_prev_region(r), r);
+ } else {
+ first->ar.start = ALIGN_DOWN(br->start,
+ DAMON_MIN_REGION);
+ last->ar.end = ALIGN(br->end, DAMON_MIN_REGION);
+ }
+ }
+}
+
+/*
+ * Update regions for current memory mappings
+ */
+void damon_va_update_regions(struct damon_ctx *ctx)
+{
+ struct damon_addr_range three_regions[3];
+ struct damon_target *t;
+
+ damon_for_each_target(t, ctx) {
+ if (damon_va_three_regions(t, three_regions))
+ continue;
+ damon_va_apply_three_regions(ctx, t, three_regions);
+ }
+}
+
+static void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm,
+ unsigned long addr)
+{
+ bool referenced = false;
+ struct page *page = pte_page(*pte);
+
+ if (pte_young(*pte)) {
+ referenced = true;
+ *pte = pte_mkold(*pte);
+ }
+
+#ifdef CONFIG_MMU_NOTIFIER
+ if (mmu_notifier_clear_young(mm, addr, addr + PAGE_SIZE))
+ referenced = true;
+#endif /* CONFIG_MMU_NOTIFIER */
+
+ if (referenced)
+ set_page_young(page);
+
+ set_page_idle(page);
+}
+
+static void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm,
+ unsigned long addr)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ bool referenced = false;
+ struct page *page = pmd_page(*pmd);
+
+ if (pmd_young(*pmd)) {
+ referenced = true;
+ *pmd = pmd_mkold(*pmd);
+ }
+
+#ifdef CONFIG_MMU_NOTIFIER
+ if (mmu_notifier_clear_young(mm, addr,
+ addr + ((1UL) << HPAGE_PMD_SHIFT)))
+ referenced = true;
+#endif /* CONFIG_MMU_NOTIFIER */
+
+ if (referenced)
+ set_page_young(page);
+
+ set_page_idle(page);
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+}
+
+static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)
+{
+ pte_t *pte = NULL;
+ pmd_t *pmd = NULL;
+ spinlock_t *ptl;
+
+ if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
+ return;
+
+ if (pte) {
+ damon_ptep_mkold(pte, mm, addr);
+ pte_unmap_unlock(pte, ptl);
+ } else {
+ damon_pmdp_mkold(pmd, mm, addr);
+ spin_unlock(ptl);
+ }
+}
+
+/*
+ * Functions for the access checking of the regions
+ */
+
+static void damon_va_prepare_access_check(struct damon_ctx *ctx,
+ struct mm_struct *mm, struct damon_region *r)
+{
+ r->sampling_addr = damon_rand(r->ar.start, r->ar.end);
+
+ damon_va_mkold(mm, r->sampling_addr);
+}
+
+void damon_va_prepare_access_checks(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct mm_struct *mm;
+ struct damon_region *r;
+
+ damon_for_each_target(t, ctx) {
+ mm = damon_get_mm(t);
+ if (!mm)
+ continue;
+ damon_for_each_region(r, t)
+ damon_va_prepare_access_check(ctx, mm, r);
+ mmput(mm);
+ }
+}
+
+static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
+ unsigned long *page_sz)
+{
+ pte_t *pte = NULL;
+ pmd_t *pmd = NULL;
+ spinlock_t *ptl;
+ bool young = false;
+
+ if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
+ return false;
+
+ *page_sz = PAGE_SIZE;
+ if (pte) {
+ young = pte_young(*pte);
+ if (!young)
+ young = !page_is_idle(pte_page(*pte));
+ pte_unmap_unlock(pte, ptl);
+ return young;
+ }
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ young = pmd_young(*pmd);
+ if (!young)
+ young = !page_is_idle(pmd_page(*pmd));
+ spin_unlock(ptl);
+ *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+ return young;
+}
+
+/*
+ * Check whether the region was accessed after the last preparation
+ *
+ * mm 'mm_struct' for the given virtual address space
+ * r the region to be checked
+ */
+static void damon_va_check_access(struct damon_ctx *ctx,
+ struct mm_struct *mm, struct damon_region *r)
+{
+ static struct mm_struct *last_mm;
+ static unsigned long last_addr;
+ static unsigned long last_page_sz = PAGE_SIZE;
+ static bool last_accessed;
+
+ /* If the region is in the last checked page, reuse the result */
+ if (mm == last_mm && (ALIGN_DOWN(last_addr, last_page_sz) ==
+ ALIGN_DOWN(r->sampling_addr, last_page_sz))) {
+ if (last_accessed)
+ r->nr_accesses++;
+ return;
+ }
+
+ last_accessed = damon_va_young(mm, r->sampling_addr, &last_page_sz);
+ if (last_accessed)
+ r->nr_accesses++;
+
+ last_mm = mm;
+ last_addr = r->sampling_addr;
+}
+
+unsigned int damon_va_check_accesses(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct mm_struct *mm;
+ struct damon_region *r;
+ unsigned int max_nr_accesses = 0;
+
+ damon_for_each_target(t, ctx) {
+ mm = damon_get_mm(t);
+ if (!mm)
+ continue;
+ damon_for_each_region(r, t) {
+ damon_va_check_access(ctx, mm, r);
+ max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
+ }
+ mmput(mm);
+ }
+
+ return max_nr_accesses;
+}
+
+/*
+ * Functions for the target validity check and cleanup
+ */
+
+bool damon_va_target_valid(void *target)
+{
+ struct damon_target *t = target;
+ struct task_struct *task;
+
+ task = damon_get_task_struct(t);
+ if (task) {
+ put_task_struct(task);
+ return true;
+ }
+
+ return false;
+}
+
+void damon_va_cleanup(struct damon_ctx *ctx)
+{
+ struct damon_target *t, *next;
+
+ damon_for_each_target_safe(t, next, ctx) {
+ put_pid((struct pid *)t->id);
+ damon_destroy_target(t);
+ }
+}
+
+void damon_va_set_primitives(struct damon_ctx *ctx)
+{
+ ctx->primitive.init_target_regions = damon_va_init_regions;
+ ctx->primitive.update_target_regions = damon_va_update_regions;
+ ctx->primitive.prepare_access_checks = damon_va_prepare_access_checks;
+ ctx->primitive.check_accesses = damon_va_check_accesses;
+ ctx->primitive.reset_aggregated = NULL;
+ ctx->primitive.target_valid = damon_va_target_valid;
+ ctx->primitive.cleanup = damon_va_cleanup;
+}
--
2.17.1
From: SeongJae Park <[email protected]>
PG_idle and PG_young allow the two PTE Accessed bit users, Idle Page
Tracking and the reclaim logic concurrently work while don't interfere
each other. That is, when they need to clear the Accessed bit, they set
PG_young to represent the previous state of the bit, respectively. And
when they need to read the bit, if the bit is cleared, they further read
the PG_young to know whether the other has cleared the bit meanwhile or
not.
We could add another page flag and extend the mechanism to use the flag
if we need to add another concurrent PTE Accessed bit user subsystem.
However, the space is limited. Meanwhile, if the new subsystem is
mutually exclusive with IDLE_PAGE_TRACKING or interfering with it is not
a real problem, it would be ok to simply reuse the PG_idle flag.
However, it's impossible because the flags are dependent on
IDLE_PAGE_TRACKING.
To allow such reuse of the flags, this commit separates the PG_young and
PG_idle flag logic from IDLE_PAGE_TRACKING and introduces new kernel
config, 'PAGE_IDLE_FLAG'. Hence, a new subsystem would be able to reuse
PG_idle without depending on IDLE_PAGE_TRACKING.
In the next commit, DAMON's reference implementation of the virtual
memory address space monitoring primitives will use it.
Signed-off-by: SeongJae Park <[email protected]>
---
include/linux/page-flags.h | 4 ++--
include/linux/page_ext.h | 2 +-
include/linux/page_idle.h | 6 +++---
include/trace/events/mmflags.h | 2 +-
mm/Kconfig | 8 ++++++++
mm/page_ext.c | 12 +++++++++++-
mm/page_idle.c | 10 ----------
7 files changed, 26 insertions(+), 18 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 4f6ba9379112..0f010fc7f1c4 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -132,7 +132,7 @@ enum pageflags {
#ifdef CONFIG_MEMORY_FAILURE
PG_hwpoison, /* hardware poisoned page. Don't touch */
#endif
-#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
+#if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
PG_young,
PG_idle,
#endif
@@ -437,7 +437,7 @@ PAGEFLAG_FALSE(HWPoison)
#define __PG_HWPOISON 0
#endif
-#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
+#if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
TESTPAGEFLAG(Young, young, PF_ANY)
SETPAGEFLAG(Young, young, PF_ANY)
TESTCLEARFLAG(Young, young, PF_ANY)
diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index cfce186f0c4e..c9cbc9756011 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -19,7 +19,7 @@ struct page_ext_operations {
enum page_ext_flags {
PAGE_EXT_OWNER,
PAGE_EXT_OWNER_ALLOCATED,
-#if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT)
+#if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT)
PAGE_EXT_YOUNG,
PAGE_EXT_IDLE,
#endif
diff --git a/include/linux/page_idle.h b/include/linux/page_idle.h
index 1e894d34bdce..d8a6aecf99cb 100644
--- a/include/linux/page_idle.h
+++ b/include/linux/page_idle.h
@@ -6,7 +6,7 @@
#include <linux/page-flags.h>
#include <linux/page_ext.h>
-#ifdef CONFIG_IDLE_PAGE_TRACKING
+#ifdef CONFIG_PAGE_IDLE_FLAG
#ifdef CONFIG_64BIT
static inline bool page_is_young(struct page *page)
@@ -106,7 +106,7 @@ static inline void clear_page_idle(struct page *page)
}
#endif /* CONFIG_64BIT */
-#else /* !CONFIG_IDLE_PAGE_TRACKING */
+#else /* !CONFIG_PAGE_IDLE_FLAG */
static inline bool page_is_young(struct page *page)
{
@@ -135,6 +135,6 @@ static inline void clear_page_idle(struct page *page)
{
}
-#endif /* CONFIG_IDLE_PAGE_TRACKING */
+#endif /* CONFIG_PAGE_IDLE_FLAG */
#endif /* _LINUX_MM_PAGE_IDLE_H */
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 67018d367b9f..ebee94c397a6 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -73,7 +73,7 @@
#define IF_HAVE_PG_HWPOISON(flag,string)
#endif
-#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
+#if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
#define IF_HAVE_PG_IDLE(flag,string) ,{1UL << flag, string}
#else
#define IF_HAVE_PG_IDLE(flag,string)
diff --git a/mm/Kconfig b/mm/Kconfig
index b97f2e8ab83f..500c885eb9f1 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -749,10 +749,18 @@ config DEFERRED_STRUCT_PAGE_INIT
lifetime of the system until these kthreads finish the
initialisation.
+config PAGE_IDLE_FLAG
+ bool "Add PG_idle and PG_young flags"
+ help
+ This feature adds PG_idle and PG_young flags in 'struct page'. PTE
+ Accessed bit writers can set the state of the bit in the flags to let
+ other PTE Accessed bit readers don't disturbed.
+
config IDLE_PAGE_TRACKING
bool "Enable idle page tracking"
depends on SYSFS && MMU
select PAGE_EXTENSION if !64BIT
+ select PAGE_IDLE_FLAG
help
This feature allows to estimate the amount of user pages that have
not been touched during a given period of time. This information can
diff --git a/mm/page_ext.c b/mm/page_ext.c
index a3616f7a0e9e..f9a6ff65ac0a 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -58,11 +58,21 @@
* can utilize this callback to initialize the state of it correctly.
*/
+#if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT)
+static bool need_page_idle(void)
+{
+ return true;
+}
+struct page_ext_operations page_idle_ops = {
+ .need = need_page_idle,
+};
+#endif
+
static struct page_ext_operations *page_ext_ops[] = {
#ifdef CONFIG_PAGE_OWNER
&page_owner_ops,
#endif
-#if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT)
+#if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT)
&page_idle_ops,
#endif
};
diff --git a/mm/page_idle.c b/mm/page_idle.c
index 057c61df12db..144fb4ed961d 100644
--- a/mm/page_idle.c
+++ b/mm/page_idle.c
@@ -211,16 +211,6 @@ static const struct attribute_group page_idle_attr_group = {
.name = "page_idle",
};
-#ifndef CONFIG_64BIT
-static bool need_page_idle(void)
-{
- return true;
-}
-struct page_ext_operations page_idle_ops = {
- .need = need_page_idle,
-};
-#endif
-
static int __init page_idle_init(void)
{
int err;
--
2.17.1
From: SeongJae Park <[email protected]>
The user space users can control DAMON and get the monitoring results
via the 'damon_aggregated' tracepoint event. However, dealing with the
tracepoint might be complex for some simple use cases. This commit
therefore implements 'recording' feature in 'damon-dbgfs'. The feature
can be used via 'record' file in the '<debugfs>/damon/' directory.
The file allows users to record monitored access patterns in a regular
binary file. The recorded results are first written in an in-memory
buffer and flushed to a file in batch. Users can get and set the size
of the buffer and the path to the result file by reading from and
writing to the ``record`` file. For example, below commands set the
buffer to be 4 KiB and the result to be saved in ``/damon.data``. ::
# cd <debugfs>/damon
# echo "4096 /damon.data" > record
# cat record
4096 /damon.data
The recording can be disabled by setting the buffer size zero.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/dbgfs.c | 261 ++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 259 insertions(+), 2 deletions(-)
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
index fd1665a183c2..733d2f3e646e 100644
--- a/mm/damon/dbgfs.c
+++ b/mm/damon/dbgfs.c
@@ -15,6 +15,17 @@
#include <linux/page_idle.h>
#include <linux/slab.h>
+#define MIN_RECORD_BUFFER_LEN 1024
+#define MAX_RECORD_BUFFER_LEN (4 * 1024 * 1024)
+#define MAX_RFILE_PATH_LEN 256
+
+struct dbgfs_recorder {
+ unsigned char *rbuf;
+ unsigned int rbuf_len;
+ unsigned int rbuf_offset;
+ char *rfile_path;
+};
+
static struct damon_ctx **dbgfs_ctxs;
static int dbgfs_nr_ctxs;
static struct dentry **dbgfs_dirs;
@@ -97,6 +108,116 @@ static ssize_t dbgfs_attrs_write(struct file *file,
return ret;
}
+static ssize_t dbgfs_record_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ struct dbgfs_recorder *rec = ctx->callback.private;
+ char record_buf[20 + MAX_RFILE_PATH_LEN];
+ int ret;
+
+ mutex_lock(&ctx->kdamond_lock);
+ ret = scnprintf(record_buf, ARRAY_SIZE(record_buf), "%u %s\n",
+ rec->rbuf_len, rec->rfile_path);
+ mutex_unlock(&ctx->kdamond_lock);
+ return simple_read_from_buffer(buf, count, ppos, record_buf, ret);
+}
+
+/*
+ * dbgfs_set_recording() - Set attributes for the recording.
+ * @ctx: target kdamond context
+ * @rbuf_len: length of the result buffer
+ * @rfile_path: path to the monitor result files
+ *
+ * Setting 'rbuf_len' 0 disables recording.
+ *
+ * This function should not be called while the kdamond is running.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+static int dbgfs_set_recording(struct damon_ctx *ctx,
+ unsigned int rbuf_len, char *rfile_path)
+{
+ struct dbgfs_recorder *recorder;
+ size_t rfile_path_len;
+
+ if (rbuf_len && (rbuf_len > MAX_RECORD_BUFFER_LEN ||
+ rbuf_len < MIN_RECORD_BUFFER_LEN)) {
+ pr_err("result buffer size (%u) is out of [%d,%d]\n",
+ rbuf_len, MIN_RECORD_BUFFER_LEN,
+ MAX_RECORD_BUFFER_LEN);
+ return -EINVAL;
+ }
+ rfile_path_len = strnlen(rfile_path, MAX_RFILE_PATH_LEN);
+ if (rfile_path_len >= MAX_RFILE_PATH_LEN) {
+ pr_err("too long (>%d) result file path %s\n",
+ MAX_RFILE_PATH_LEN, rfile_path);
+ return -EINVAL;
+ }
+
+ recorder = ctx->callback.private;
+ if (!recorder) {
+ recorder = kzalloc(sizeof(*recorder), GFP_KERNEL);
+ if (!recorder)
+ return -ENOMEM;
+ ctx->callback.private = recorder;
+ }
+
+ recorder->rbuf_len = rbuf_len;
+ kfree(recorder->rbuf);
+ recorder->rbuf = NULL;
+ kfree(recorder->rfile_path);
+ recorder->rfile_path = NULL;
+
+ if (rbuf_len) {
+ recorder->rbuf = kvmalloc(rbuf_len, GFP_KERNEL);
+ if (!recorder->rbuf)
+ return -ENOMEM;
+ }
+ recorder->rfile_path = kmalloc(rfile_path_len + 1, GFP_KERNEL);
+ if (!recorder->rfile_path)
+ return -ENOMEM;
+ strncpy(recorder->rfile_path, rfile_path, rfile_path_len + 1);
+
+ return 0;
+}
+
+static ssize_t dbgfs_record_write(struct file *file,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf;
+ unsigned int rbuf_len;
+ char rfile_path[MAX_RFILE_PATH_LEN];
+ ssize_t ret = count;
+ int err;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+
+ if (sscanf(kbuf, "%u %s",
+ &rbuf_len, rfile_path) != 2) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ ret = -EBUSY;
+ goto unlock_out;
+ }
+
+ err = dbgfs_set_recording(ctx, rbuf_len, rfile_path);
+ if (err)
+ ret = err;
+unlock_out:
+ mutex_unlock(&ctx->kdamond_lock);
+out:
+ kfree(kbuf);
+ return ret;
+}
+
#define targetid_is_pid(ctx) \
(ctx->primitive.target_valid == damon_va_target_valid)
@@ -230,6 +351,13 @@ static const struct file_operations attrs_fops = {
.write = dbgfs_attrs_write,
};
+static const struct file_operations record_fops = {
+ .owner = THIS_MODULE,
+ .open = damon_dbgfs_open,
+ .read = dbgfs_record_read,
+ .write = dbgfs_record_write,
+};
+
static const struct file_operations target_ids_fops = {
.owner = THIS_MODULE,
.open = damon_dbgfs_open,
@@ -239,8 +367,9 @@ static const struct file_operations target_ids_fops = {
static int dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx)
{
- const char * const file_names[] = {"attrs", "target_ids"};
- const struct file_operations *fops[] = {&attrs_fops, &target_ids_fops};
+ const char * const file_names[] = {"attrs", "record", "target_ids"};
+ const struct file_operations *fops[] = {&attrs_fops, &record_fops,
+ &target_ids_fops};
int i;
for (i = 0; i < ARRAY_SIZE(file_names); i++) {
@@ -254,6 +383,126 @@ static int dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx)
return 0;
}
+/*
+ * Flush the content in the result buffer to the result file
+ */
+static void dbgfs_flush_rbuffer(struct dbgfs_recorder *rec)
+{
+ ssize_t sz;
+ loff_t pos = 0;
+ struct file *rfile;
+
+ if (!rec->rbuf_offset)
+ return;
+
+ rfile = filp_open(rec->rfile_path,
+ O_CREAT | O_RDWR | O_APPEND | O_LARGEFILE, 0644);
+ if (IS_ERR(rfile)) {
+ pr_err("Cannot open the result file %s\n",
+ rec->rfile_path);
+ return;
+ }
+
+ while (rec->rbuf_offset) {
+ sz = kernel_write(rfile, rec->rbuf, rec->rbuf_offset, &pos);
+ if (sz < 0)
+ break;
+ rec->rbuf_offset -= sz;
+ }
+ filp_close(rfile, NULL);
+}
+
+/*
+ * Write a data into the result buffer
+ */
+static void dbgfs_write_rbuf(struct damon_ctx *ctx, void *data, ssize_t size)
+{
+ struct dbgfs_recorder *rec = ctx->callback.private;
+
+ if (!rec->rbuf_len || !rec->rbuf || !rec->rfile_path)
+ return;
+ if (rec->rbuf_offset + size > rec->rbuf_len)
+ dbgfs_flush_rbuffer(ctx->callback.private);
+ if (rec->rbuf_offset + size > rec->rbuf_len) {
+ pr_warn("%s: flush failed, or wrong size given(%u, %zu)\n",
+ __func__, rec->rbuf_offset, size);
+ return;
+ }
+
+ memcpy(&rec->rbuf[rec->rbuf_offset], data, size);
+ rec->rbuf_offset += size;
+}
+
+static void dbgfs_write_record_header(struct damon_ctx *ctx)
+{
+ int recfmt_ver = 2;
+
+ dbgfs_write_rbuf(ctx, "damon_recfmt_ver", 16);
+ dbgfs_write_rbuf(ctx, &recfmt_ver, sizeof(recfmt_ver));
+}
+
+static unsigned int nr_damon_targets(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ unsigned int nr_targets = 0;
+
+ damon_for_each_target(t, ctx)
+ nr_targets++;
+
+ return nr_targets;
+}
+
+static int dbgfs_before_start(struct damon_ctx *ctx)
+{
+ dbgfs_write_record_header(ctx);
+ return 0;
+}
+
+/*
+ * Store the aggregated monitoring results to the result buffer
+ *
+ * The format for the result buffer is as below:
+ *
+ * <time> <number of targets> <array of target infos>
+ *
+ * target info: <id> <number of regions> <array of region infos>
+ * region info: <start address> <end address> <nr_accesses>
+ */
+static int dbgfs_after_aggregation(struct damon_ctx *c)
+{
+ struct damon_target *t;
+ struct timespec64 now;
+ unsigned int nr;
+
+ ktime_get_coarse_ts64(&now);
+
+ dbgfs_write_rbuf(c, &now, sizeof(now));
+ nr = nr_damon_targets(c);
+ dbgfs_write_rbuf(c, &nr, sizeof(nr));
+
+ damon_for_each_target(t, c) {
+ struct damon_region *r;
+
+ dbgfs_write_rbuf(c, &t->id, sizeof(t->id));
+ nr = damon_nr_regions(t);
+ dbgfs_write_rbuf(c, &nr, sizeof(nr));
+ damon_for_each_region(r, t) {
+ dbgfs_write_rbuf(c, &r->ar.start, sizeof(r->ar.start));
+ dbgfs_write_rbuf(c, &r->ar.end, sizeof(r->ar.end));
+ dbgfs_write_rbuf(c, &r->nr_accesses,
+ sizeof(r->nr_accesses));
+ }
+ }
+
+ return 0;
+}
+
+static int dbgfs_before_terminate(struct damon_ctx *ctx)
+{
+ dbgfs_flush_rbuffer(ctx->callback.private);
+ return 0;
+}
+
static struct damon_ctx *dbgfs_new_ctx(void)
{
struct damon_ctx *ctx;
@@ -262,7 +511,15 @@ static struct damon_ctx *dbgfs_new_ctx(void)
if (!ctx)
return NULL;
+ if (dbgfs_set_recording(ctx, 0, "none")) {
+ damon_destroy_ctx(ctx);
+ return NULL;
+ }
+
damon_va_set_primitives(ctx);
+ ctx->callback.before_start = dbgfs_before_start;
+ ctx->callback.after_aggregation = dbgfs_after_aggregation;
+ ctx->callback.before_terminate = dbgfs_before_terminate;
return ctx;
}
--
2.17.1
From: SeongJae Park <[email protected]>
DAMON is designed to be used by kernel space code such as the memory
management subsystems, and therefore it provides only kernel space API.
That said, letting the user space control DAMON could provide some
benefits to them. For example, it will allow user space to analyze
their specific workloads and make their own special optimizations.
For such cases, this commit implements a simple DAMON application kernel
module, namely 'damon-dbgfs', which merely wraps the DAMON api and
exports those to the user space via the debugfs.
'damon-dbgfs' exports three files, ``attrs``, ``target_ids``, and
``monitor_on`` under its debugfs directory, ``<debugfs>/damon/``.
Attributes
----------
Users can read and write the ``sampling interval``, ``aggregation
interval``, ``regions update interval``, and min/max number of
monitoring target regions by reading from and writing to the ``attrs``
file. For example, below commands set those values to 5 ms, 100 ms,
1,000 ms, 10, 1000 and check it again::
# cd <debugfs>/damon
# echo 5000 100000 1000000 10 1000 > attrs
# cat attrs
5000 100000 1000000 10 1000
Target IDs
----------
Some types of address spaces supports multiple monitoring target. For
example, the virtual memory address spaces monitoring can have multiple
processes as the monitoring targets. Users can set the targets by
writing relevant id values of the targets to, and get the ids of the
current targets by reading from the ``target_ids`` file. In case of the
virtual address spaces monitoring, the values should be pids of the
monitoring target processes. For example, below commands set processes
having pids 42 and 4242 as the monitoring targets and check it again::
# cd <debugfs>/damon
# echo 42 4242 > target_ids
# cat target_ids
42 4242
Note that setting the target ids doesn't start the monitoring.
Turning On/Off
--------------
Setting the files as described above doesn't incur effect unless you
explicitly start the monitoring. You can start, stop, and check the
current status of the monitoring by writing to and reading from the
``monitor_on`` file. Writing ``on`` to the file starts the monitoring
of the targets with the attributes. Writing ``off`` to the file stops
those. DAMON also stops if every targets are invalidated (in case of
the virtual memory monitoring, target processes are invalidated when
terminated). Below example commands turn on, off, and check the status
of DAMON::
# cd <debugfs>/damon
# echo on > monitor_on
# echo off > monitor_on
# cat monitor_on
off
Please note that you cannot write to the above-mentioned debugfs files
while the monitoring is turned on. If you write to the files while
DAMON is running, an error code such as ``-EBUSY`` will be returned.
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Leonard Foerster <[email protected]>
---
include/linux/damon.h | 3 +
mm/damon/Kconfig | 9 ++
mm/damon/Makefile | 1 +
mm/damon/core.c | 45 ++++++
mm/damon/dbgfs.c | 366 ++++++++++++++++++++++++++++++++++++++++++
5 files changed, 424 insertions(+)
create mode 100644 mm/damon/dbgfs.c
diff --git a/include/linux/damon.h b/include/linux/damon.h
index 39b4d6d3ddee..f9e0d4349352 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -265,9 +265,12 @@ unsigned int damon_nr_regions(struct damon_target *t);
struct damon_ctx *damon_new_ctx(enum damon_target_type type);
void damon_destroy_ctx(struct damon_ctx *ctx);
+int damon_set_targets(struct damon_ctx *ctx,
+ unsigned long *ids, ssize_t nr_ids);
int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
unsigned long aggr_int, unsigned long regions_update_int,
unsigned long min_nr_reg, unsigned long max_nr_reg);
+int damon_nr_running_ctxs(void);
int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
index 8ae080c52950..72f1683ba0ee 100644
--- a/mm/damon/Kconfig
+++ b/mm/damon/Kconfig
@@ -21,4 +21,13 @@ config DAMON_VADDR
This builds the default data access monitoring primitives for DAMON
that works for virtual address spaces.
+config DAMON_DBGFS
+ bool "DAMON debugfs interface"
+ depends on DAMON_VADDR && DEBUG_FS
+ help
+ This builds the debugfs interface for DAMON. The user space admins
+ can use the interface for arbitrary data access monitoring.
+
+ If unsure, say N.
+
endmenu
diff --git a/mm/damon/Makefile b/mm/damon/Makefile
index 6ebbd08aed67..fed4be3bace3 100644
--- a/mm/damon/Makefile
+++ b/mm/damon/Makefile
@@ -2,3 +2,4 @@
obj-$(CONFIG_DAMON) := core.o
obj-$(CONFIG_DAMON_VADDR) += vaddr.o
+obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 5ca9f79ccbb6..b9575a6bebff 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -166,6 +166,37 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
kfree(ctx);
}
+/**
+ * damon_set_targets() - Set monitoring targets.
+ * @ctx: monitoring context
+ * @ids: array of target ids
+ * @nr_ids: number of entries in @ids
+ *
+ * This function should not be called while the kdamond is running.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int damon_set_targets(struct damon_ctx *ctx,
+ unsigned long *ids, ssize_t nr_ids)
+{
+ ssize_t i;
+ struct damon_target *t, *next;
+
+ damon_for_each_target_safe(t, next, ctx)
+ damon_destroy_target(t);
+
+ for (i = 0; i < nr_ids; i++) {
+ t = damon_new_target(ids[i]);
+ if (!t) {
+ pr_err("Failed to alloc damon_target\n");
+ return -ENOMEM;
+ }
+ damon_add_target(ctx, t);
+ }
+
+ return 0;
+}
+
/**
* damon_set_attrs() - Set attributes for the monitoring.
* @ctx: monitoring context
@@ -206,6 +237,20 @@ int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
return 0;
}
+/**
+ * damon_nr_running_ctxs() - Return number of currently running contexts.
+ */
+int damon_nr_running_ctxs(void)
+{
+ int nr_ctxs;
+
+ mutex_lock(&damon_lock);
+ nr_ctxs = nr_running_ctxs;
+ mutex_unlock(&damon_lock);
+
+ return nr_ctxs;
+}
+
/* Returns the size upper limit for each monitoring region */
static unsigned long damon_region_sz_limit(struct damon_ctx *ctx)
{
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
new file mode 100644
index 000000000000..fd1665a183c2
--- /dev/null
+++ b/mm/damon/dbgfs.c
@@ -0,0 +1,366 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DAMON Debugfs Interface
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#define pr_fmt(fmt) "damon-dbgfs: " fmt
+
+#include <linux/damon.h>
+#include <linux/debugfs.h>
+#include <linux/file.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/page_idle.h>
+#include <linux/slab.h>
+
+static struct damon_ctx **dbgfs_ctxs;
+static int dbgfs_nr_ctxs;
+static struct dentry **dbgfs_dirs;
+
+/*
+ * Returns non-empty string on success, negarive error code otherwise.
+ */
+static char *user_input_str(const char __user *buf, size_t count, loff_t *ppos)
+{
+ char *kbuf;
+ ssize_t ret;
+
+ /* We do not accept continuous write */
+ if (*ppos)
+ return ERR_PTR(-EINVAL);
+
+ kbuf = kmalloc(count + 1, GFP_KERNEL);
+ if (!kbuf)
+ return ERR_PTR(-ENOMEM);
+
+ ret = simple_write_to_buffer(kbuf, count + 1, ppos, buf, count);
+ if (ret != count) {
+ kfree(kbuf);
+ return ERR_PTR(-EIO);
+ }
+ kbuf[ret] = '\0';
+
+ return kbuf;
+}
+
+static ssize_t dbgfs_attrs_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char kbuf[128];
+ int ret;
+
+ mutex_lock(&ctx->kdamond_lock);
+ ret = scnprintf(kbuf, ARRAY_SIZE(kbuf), "%lu %lu %lu %lu %lu\n",
+ ctx->sample_interval, ctx->aggr_interval,
+ ctx->regions_update_interval, ctx->min_nr_regions,
+ ctx->max_nr_regions);
+ mutex_unlock(&ctx->kdamond_lock);
+
+ return simple_read_from_buffer(buf, count, ppos, kbuf, ret);
+}
+
+static ssize_t dbgfs_attrs_write(struct file *file,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ unsigned long s, a, r, minr, maxr;
+ char *kbuf;
+ ssize_t ret = count;
+ int err;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+
+ if (sscanf(kbuf, "%lu %lu %lu %lu %lu",
+ &s, &a, &r, &minr, &maxr) != 5) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ ret = -EBUSY;
+ goto unlock_out;
+ }
+
+ err = damon_set_attrs(ctx, s, a, r, minr, maxr);
+ if (err)
+ ret = err;
+unlock_out:
+ mutex_unlock(&ctx->kdamond_lock);
+out:
+ kfree(kbuf);
+ return ret;
+}
+
+#define targetid_is_pid(ctx) \
+ (ctx->primitive.target_valid == damon_va_target_valid)
+
+static ssize_t sprint_target_ids(struct damon_ctx *ctx, char *buf, ssize_t len)
+{
+ struct damon_target *t;
+ unsigned long id;
+ int written = 0;
+ int rc;
+
+ damon_for_each_target(t, ctx) {
+ id = t->id;
+ if (targetid_is_pid(ctx))
+ /* Show pid numbers to debugfs users */
+ id = (unsigned long)pid_vnr((struct pid *)id);
+
+ rc = scnprintf(&buf[written], len - written, "%lu ", id);
+ if (!rc)
+ return -ENOMEM;
+ written += rc;
+ }
+ if (written)
+ written -= 1;
+ written += scnprintf(&buf[written], len - written, "\n");
+ return written;
+}
+
+static ssize_t dbgfs_target_ids_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ ssize_t len;
+ char ids_buf[320];
+
+ mutex_lock(&ctx->kdamond_lock);
+ len = sprint_target_ids(ctx, ids_buf, 320);
+ mutex_unlock(&ctx->kdamond_lock);
+ if (len < 0)
+ return len;
+
+ return simple_read_from_buffer(buf, count, ppos, ids_buf, len);
+}
+
+/*
+ * Converts a string into an array of unsigned long integers
+ *
+ * Returns an array of unsigned long integers if the conversion success, or
+ * NULL otherwise.
+ */
+static unsigned long *str_to_target_ids(const char *str, ssize_t len,
+ ssize_t *nr_ids)
+{
+ unsigned long *ids;
+ const int max_nr_ids = 32;
+ unsigned long id;
+ int pos = 0, parsed, ret;
+
+ *nr_ids = 0;
+ ids = kmalloc_array(max_nr_ids, sizeof(id), GFP_KERNEL);
+ if (!ids)
+ return NULL;
+ while (*nr_ids < max_nr_ids && pos < len) {
+ ret = sscanf(&str[pos], "%lu%n", &id, &parsed);
+ pos += parsed;
+ if (ret != 1)
+ break;
+ ids[*nr_ids] = id;
+ *nr_ids += 1;
+ }
+
+ return ids;
+}
+
+static ssize_t dbgfs_target_ids_write(struct file *file,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf, *nrs;
+ unsigned long *targets;
+ ssize_t nr_targets;
+ ssize_t ret = count;
+ int i;
+ int err;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+
+ nrs = kbuf;
+
+ targets = str_to_target_ids(nrs, ret, &nr_targets);
+ if (!targets) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ if (targetid_is_pid(ctx)) {
+ for (i = 0; i < nr_targets; i++)
+ targets[i] = (unsigned long)find_get_pid(
+ (int)targets[i]);
+ }
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ ret = -EINVAL;
+ goto unlock_out;
+ }
+
+ err = damon_set_targets(ctx, targets, nr_targets);
+ if (err)
+ ret = err;
+unlock_out:
+ mutex_unlock(&ctx->kdamond_lock);
+ kfree(targets);
+out:
+ kfree(kbuf);
+ return ret;
+}
+
+static int damon_dbgfs_open(struct inode *inode, struct file *file)
+{
+ file->private_data = inode->i_private;
+
+ return nonseekable_open(inode, file);
+}
+
+static const struct file_operations attrs_fops = {
+ .owner = THIS_MODULE,
+ .open = damon_dbgfs_open,
+ .read = dbgfs_attrs_read,
+ .write = dbgfs_attrs_write,
+};
+
+static const struct file_operations target_ids_fops = {
+ .owner = THIS_MODULE,
+ .open = damon_dbgfs_open,
+ .read = dbgfs_target_ids_read,
+ .write = dbgfs_target_ids_write,
+};
+
+static int dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx)
+{
+ const char * const file_names[] = {"attrs", "target_ids"};
+ const struct file_operations *fops[] = {&attrs_fops, &target_ids_fops};
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(file_names); i++) {
+ if (!debugfs_create_file(file_names[i], 0600, dir,
+ ctx, fops[i])) {
+ pr_err("failed to create %s file\n", file_names[i]);
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+static struct damon_ctx *dbgfs_new_ctx(void)
+{
+ struct damon_ctx *ctx;
+
+ ctx = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ if (!ctx)
+ return NULL;
+
+ damon_va_set_primitives(ctx);
+ return ctx;
+}
+
+static ssize_t dbgfs_monitor_on_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos)
+{
+ char monitor_on_buf[5];
+ bool monitor_on = damon_nr_running_ctxs() != 0;
+ int len;
+
+ len = scnprintf(monitor_on_buf, 5, monitor_on ? "on\n" : "off\n");
+
+ return simple_read_from_buffer(buf, count, ppos, monitor_on_buf, len);
+}
+
+static ssize_t dbgfs_monitor_on_write(struct file *file,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ ssize_t ret = count;
+ char *kbuf;
+ int err;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+
+ /* Remove white space */
+ if (sscanf(kbuf, "%s", kbuf) != 1) {
+ kfree(kbuf);
+ return -EINVAL;
+ }
+
+ if (!strncmp(kbuf, "on", count))
+ err = damon_start(dbgfs_ctxs, dbgfs_nr_ctxs);
+ else if (!strncmp(kbuf, "off", count))
+ err = damon_stop(dbgfs_ctxs, dbgfs_nr_ctxs);
+ else
+ err = -EINVAL;
+
+ if (err)
+ ret = err;
+ kfree(kbuf);
+ return ret;
+}
+
+static const struct file_operations monitor_on_fops = {
+ .owner = THIS_MODULE,
+ .read = dbgfs_monitor_on_read,
+ .write = dbgfs_monitor_on_write,
+};
+
+static int __init __damon_dbgfs_init(void)
+{
+ struct dentry *dbgfs_root;
+ const char * const file_names[] = {"monitor_on"};
+ const struct file_operations *fops[] = {&monitor_on_fops};
+ int i;
+
+ dbgfs_root = debugfs_create_dir("damon", NULL);
+ if (IS_ERR(dbgfs_root)) {
+ pr_err("failed to create the dbgfs dir\n");
+ return PTR_ERR(dbgfs_root);
+ }
+
+ for (i = 0; i < ARRAY_SIZE(file_names); i++) {
+ if (!debugfs_create_file(file_names[i], 0600, dbgfs_root,
+ NULL, fops[i])) {
+ pr_err("failed to create %s file\n", file_names[i]);
+ return -ENOMEM;
+ }
+ }
+ dbgfs_fill_ctx_dir(dbgfs_root, dbgfs_ctxs[0]);
+
+ dbgfs_dirs = kmalloc_array(1, sizeof(dbgfs_root), GFP_KERNEL);
+ dbgfs_dirs[0] = dbgfs_root;
+
+ return 0;
+}
+
+/*
+ * Functions for the initialization
+ */
+
+static int __init damon_dbgfs_init(void)
+{
+ int rc;
+
+ dbgfs_ctxs = kmalloc(sizeof(*dbgfs_ctxs), GFP_KERNEL);
+ dbgfs_ctxs[0] = dbgfs_new_ctx();
+ if (!dbgfs_ctxs[0])
+ return -ENOMEM;
+ dbgfs_nr_ctxs = 1;
+
+ rc = __damon_dbgfs_init();
+ if (rc)
+ pr_err("%s: dbgfs init failed\n", __func__);
+
+ return rc;
+}
+
+module_init(damon_dbgfs_init);
--
2.17.1
From: SeongJae Park <[email protected]>
For CPU usage accounting, knowing pid of the monitoring thread could be
helpful. For example, users could use cpuaccount cgroups with the pid.
This commit therefore exports the pid of currently running monitoring
thread to the user space via 'kdamond_pid' file in the debugfs
directory.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/dbgfs.c | 37 +++++++++++++++++++++++++++++++++++--
1 file changed, 35 insertions(+), 2 deletions(-)
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
index 733d2f3e646e..a1512d3e5afe 100644
--- a/mm/damon/dbgfs.c
+++ b/mm/damon/dbgfs.c
@@ -337,6 +337,32 @@ static ssize_t dbgfs_target_ids_write(struct file *file,
return ret;
}
+static ssize_t dbgfs_kdamond_pid_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf;
+ ssize_t len;
+
+ kbuf = kmalloc(count, GFP_KERNEL);
+ if (!kbuf)
+ return -ENOMEM;
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond)
+ len = scnprintf(kbuf, count, "%d\n", ctx->kdamond->pid);
+ else
+ len = scnprintf(kbuf, count, "none\n");
+ mutex_unlock(&ctx->kdamond_lock);
+ if (!len)
+ goto out;
+ len = simple_read_from_buffer(buf, count, ppos, kbuf, len);
+
+out:
+ kfree(kbuf);
+ return len;
+}
+
static int damon_dbgfs_open(struct inode *inode, struct file *file)
{
file->private_data = inode->i_private;
@@ -365,11 +391,18 @@ static const struct file_operations target_ids_fops = {
.write = dbgfs_target_ids_write,
};
+static const struct file_operations kdamond_pid_fops = {
+ .owner = THIS_MODULE,
+ .open = damon_dbgfs_open,
+ .read = dbgfs_kdamond_pid_read,
+};
+
static int dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx)
{
- const char * const file_names[] = {"attrs", "record", "target_ids"};
+ const char * const file_names[] = {"attrs", "record", "target_ids",
+ "kdamond_pid"};
const struct file_operations *fops[] = {&attrs_fops, &record_fops,
- &target_ids_fops};
+ &target_ids_fops, &kdamond_pid_fops};
int i;
for (i = 0; i < ARRAY_SIZE(file_names); i++) {
--
2.17.1
From: SeongJae Park <[email protected]>
This commit adds a tracepoint for DAMON. It traces the monitoring
results of each region for each aggregation interval. Using this, DAMON
can easily integrated with tracepoints supporting tools such as perf.
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Leonard Foerster <[email protected]>
Reviewed-by: Steven Rostedt (VMware) <[email protected]>
---
include/trace/events/damon.h | 43 ++++++++++++++++++++++++++++++++++++
mm/damon/core.c | 7 +++++-
2 files changed, 49 insertions(+), 1 deletion(-)
create mode 100644 include/trace/events/damon.h
diff --git a/include/trace/events/damon.h b/include/trace/events/damon.h
new file mode 100644
index 000000000000..2f422f4f1fb9
--- /dev/null
+++ b/include/trace/events/damon.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM damon
+
+#if !defined(_TRACE_DAMON_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_DAMON_H
+
+#include <linux/damon.h>
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(damon_aggregated,
+
+ TP_PROTO(struct damon_target *t, struct damon_region *r,
+ unsigned int nr_regions),
+
+ TP_ARGS(t, r, nr_regions),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, target_id)
+ __field(unsigned int, nr_regions)
+ __field(unsigned long, start)
+ __field(unsigned long, end)
+ __field(unsigned int, nr_accesses)
+ ),
+
+ TP_fast_assign(
+ __entry->target_id = t->id;
+ __entry->nr_regions = nr_regions;
+ __entry->start = r->ar.start;
+ __entry->end = r->ar.end;
+ __entry->nr_accesses = r->nr_accesses;
+ ),
+
+ TP_printk("target_id=%lu nr_regions=%u %lu-%lu: %u",
+ __entry->target_id, __entry->nr_regions,
+ __entry->start, __entry->end, __entry->nr_accesses)
+);
+
+#endif /* _TRACE_DAMON_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 0f9beb60d9dd..5ca9f79ccbb6 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -13,6 +13,9 @@
#include <linux/random.h>
#include <linux/slab.h>
+#define CREATE_TRACE_POINTS
+#include <trace/events/damon.h>
+
/* Get a random number in [l, r) */
#define damon_rand(l, r) (l + prandom_u32_max(r - l))
@@ -384,8 +387,10 @@ static void kdamond_reset_aggregated(struct damon_ctx *c)
damon_for_each_target(t, c) {
struct damon_region *r;
- damon_for_each_region(r, t)
+ damon_for_each_region(r, t) {
+ trace_damon_aggregated(t, r, damon_nr_regions(t));
r->nr_accesses = 0;
+ }
}
}
--
2.17.1
From: SeongJae Park <[email protected]>
In some use cases, users would want to run multiple monitoring context.
For example, if a user wants a high precision monitoring and dedicating
multiple CPUs for the job is ok, because DAMON creates one monitoring
thread per one context, the user can split the monitoring target regions
into multiple small regions and create one context for each region. Or,
someone might want to simultaneously monitor different address spaces,
e.g., both virtual address space and physical address space.
The DAMON's API allows such usage, but 'damon-dbgfs' does not.
Therefore, only kernel space DAMON users can do multiple contexts
monitoring.
This commit allows the user space DAMON users to use multiple contexts
monitoring by introducing two new 'damon-dbgfs' debugfs files,
'mk_context' and 'rm_context'. Users can create a new monitoring
context by writing the desired name of the new context to 'mk_context'.
Then, a new directory with the name and having the files for setting of
the context ('attrs', 'target_ids' and 'record') will be created under
the debugfs directory. Writing the name of the context to remove to
'rm_context' will remove the related context and directory.
Signed-off-by: SeongJae Park <[email protected]>
---
mm/damon/dbgfs.c | 213 ++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 211 insertions(+), 2 deletions(-)
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
index a1512d3e5afe..46186b057e1a 100644
--- a/mm/damon/dbgfs.c
+++ b/mm/damon/dbgfs.c
@@ -29,6 +29,7 @@ struct dbgfs_recorder {
static struct damon_ctx **dbgfs_ctxs;
static int dbgfs_nr_ctxs;
static struct dentry **dbgfs_dirs;
+static DEFINE_MUTEX(damon_dbgfs_lock);
/*
* Returns non-empty string on success, negarive error code otherwise.
@@ -474,6 +475,13 @@ static void dbgfs_write_record_header(struct damon_ctx *ctx)
dbgfs_write_rbuf(ctx, &recfmt_ver, sizeof(recfmt_ver));
}
+static void dbgfs_free_recorder(struct dbgfs_recorder *recorder)
+{
+ kfree(recorder->rbuf);
+ kfree(recorder->rfile_path);
+ kfree(recorder);
+}
+
static unsigned int nr_damon_targets(struct damon_ctx *ctx)
{
struct damon_target *t;
@@ -556,6 +564,195 @@ static struct damon_ctx *dbgfs_new_ctx(void)
return ctx;
}
+static void dbgfs_destroy_ctx(struct damon_ctx *ctx)
+{
+ dbgfs_free_recorder(ctx->callback.private);
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * Make a context of @name and create a debugfs directory for it.
+ *
+ * This function should be called while holding damon_dbgfs_lock.
+ *
+ * Returns 0 on success, negative error code otherwise.
+ */
+static int dbgfs_mk_context(char *name)
+{
+ struct dentry *root, **new_dirs, *new_dir;
+ struct damon_ctx **new_ctxs, *new_ctx;
+ int err;
+
+ if (damon_nr_running_ctxs())
+ return -EBUSY;
+
+ new_ctxs = krealloc(dbgfs_ctxs, sizeof(*dbgfs_ctxs) *
+ (dbgfs_nr_ctxs + 1), GFP_KERNEL);
+ if (!new_ctxs)
+ return -ENOMEM;
+
+ new_dirs = krealloc(dbgfs_dirs, sizeof(*dbgfs_dirs) *
+ (dbgfs_nr_ctxs + 1), GFP_KERNEL);
+ if (!new_dirs) {
+ kfree(new_ctxs);
+ return -ENOMEM;
+ }
+
+ dbgfs_ctxs = new_ctxs;
+ dbgfs_dirs = new_dirs;
+
+ root = dbgfs_dirs[0];
+ if (!root)
+ return -ENOENT;
+
+ new_dir = debugfs_create_dir(name, root);
+ if (IS_ERR(new_dir))
+ return PTR_ERR(new_dir);
+ dbgfs_dirs[dbgfs_nr_ctxs] = new_dir;
+
+ new_ctx = dbgfs_new_ctx();
+ if (!new_ctx) {
+ debugfs_remove(new_dir);
+ dbgfs_dirs[dbgfs_nr_ctxs] = NULL;
+ return -ENOMEM;
+ }
+ dbgfs_ctxs[dbgfs_nr_ctxs] = new_ctx;
+
+ err = dbgfs_fill_ctx_dir(dbgfs_dirs[dbgfs_nr_ctxs],
+ dbgfs_ctxs[dbgfs_nr_ctxs]);
+ if (err)
+ return err;
+
+ dbgfs_nr_ctxs++;
+ return 0;
+}
+
+static ssize_t dbgfs_mk_context_write(struct file *file,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ char *kbuf;
+ char *ctx_name;
+ ssize_t ret = count;
+ int err;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+ ctx_name = kmalloc(count + 1, GFP_KERNEL);
+ if (!ctx_name) {
+ kfree(kbuf);
+ return -ENOMEM;
+ }
+
+ /* Trim white space */
+ if (sscanf(kbuf, "%s", ctx_name) != 1) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ mutex_lock(&damon_dbgfs_lock);
+ err = dbgfs_mk_context(ctx_name);
+ if (err)
+ ret = err;
+ mutex_unlock(&damon_dbgfs_lock);
+
+out:
+ kfree(kbuf);
+ kfree(ctx_name);
+ return ret;
+}
+
+/*
+ * Remove a context of @name and its debugfs directory.
+ *
+ * This function should be called while holding damon_dbgfs_lock.
+ *
+ * Return 0 on success, negative error code otherwise.
+ */
+static int dbgfs_rm_context(char *name)
+{
+ struct dentry *root, *dir, **new_dirs;
+ struct damon_ctx **new_ctxs;
+ int i, j;
+
+ if (damon_nr_running_ctxs())
+ return -EBUSY;
+
+ root = dbgfs_dirs[0];
+ if (!root)
+ return -ENOENT;
+
+ dir = debugfs_lookup(name, root);
+ if (!dir)
+ return -ENOENT;
+
+ new_dirs = kmalloc_array(dbgfs_nr_ctxs - 1, sizeof(*dbgfs_dirs),
+ GFP_KERNEL);
+ if (!new_dirs)
+ return -ENOMEM;
+
+ new_ctxs = kmalloc_array(dbgfs_nr_ctxs - 1, sizeof(*dbgfs_ctxs),
+ GFP_KERNEL);
+ if (!new_ctxs) {
+ kfree(new_dirs);
+ return -ENOMEM;
+ }
+
+ for (i = 0, j = 0; i < dbgfs_nr_ctxs; i++) {
+ if (dbgfs_dirs[i] == dir) {
+ debugfs_remove(dbgfs_dirs[i]);
+ dbgfs_destroy_ctx(dbgfs_ctxs[i]);
+ continue;
+ }
+ new_dirs[j] = dbgfs_dirs[i];
+ new_ctxs[j++] = dbgfs_ctxs[i];
+ }
+
+ kfree(dbgfs_dirs);
+ kfree(dbgfs_ctxs);
+
+ dbgfs_dirs = new_dirs;
+ dbgfs_ctxs = new_ctxs;
+ dbgfs_nr_ctxs--;
+
+ return 0;
+}
+
+static ssize_t dbgfs_rm_context_write(struct file *file,
+ const char __user *buf, size_t count, loff_t *ppos)
+{
+ char *kbuf;
+ ssize_t ret = count;
+ int err;
+ char *ctx_name;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+ ctx_name = kmalloc(count + 1, GFP_KERNEL);
+ if (!ctx_name) {
+ kfree(kbuf);
+ return -ENOMEM;
+ }
+
+ /* Trim white space */
+ if (sscanf(kbuf, "%s", ctx_name) != 1) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ mutex_lock(&damon_dbgfs_lock);
+ err = dbgfs_rm_context(ctx_name);
+ if (err)
+ ret = err;
+ mutex_unlock(&damon_dbgfs_lock);
+
+out:
+ kfree(kbuf);
+ kfree(ctx_name);
+ return ret;
+}
+
static ssize_t dbgfs_monitor_on_read(struct file *file,
char __user *buf, size_t count, loff_t *ppos)
{
@@ -598,6 +795,16 @@ static ssize_t dbgfs_monitor_on_write(struct file *file,
return ret;
}
+static const struct file_operations mk_contexts_fops = {
+ .owner = THIS_MODULE,
+ .write = dbgfs_mk_context_write,
+};
+
+static const struct file_operations rm_contexts_fops = {
+ .owner = THIS_MODULE,
+ .write = dbgfs_rm_context_write,
+};
+
static const struct file_operations monitor_on_fops = {
.owner = THIS_MODULE,
.read = dbgfs_monitor_on_read,
@@ -607,8 +814,10 @@ static const struct file_operations monitor_on_fops = {
static int __init __damon_dbgfs_init(void)
{
struct dentry *dbgfs_root;
- const char * const file_names[] = {"monitor_on"};
- const struct file_operations *fops[] = {&monitor_on_fops};
+ const char * const file_names[] = {"mk_contexts", "rm_contexts",
+ "monitor_on"};
+ const struct file_operations *fops[] = {&mk_contexts_fops,
+ &rm_contexts_fops, &monitor_on_fops};
int i;
dbgfs_root = debugfs_create_dir("damon", NULL);
--
2.17.1
From: SeongJae Park <[email protected]>
This commit updates MAINTAINERS file for DAMON related files.
Signed-off-by: SeongJae Park <[email protected]>
---
MAINTAINERS | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 281de213ef47..88b2125b0f07 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4872,6 +4872,18 @@ F: net/ax25/ax25_out.c
F: net/ax25/ax25_timer.c
F: net/ax25/sysctl_net_ax25.c
+DATA ACCESS MONITOR
+M: SeongJae Park <[email protected]>
+L: [email protected]
+S: Maintained
+F: Documentation/admin-guide/mm/damon/*
+F: Documentation/vm/damon/*
+F: include/linux/damon.h
+F: include/trace/events/damon.h
+F: mm/damon/*
+F: tools/damon/*
+F: tools/testing/selftests/damon/*
+
DAVICOM FAST ETHERNET (DMFE) NETWORK DRIVER
L: [email protected]
S: Orphan
--
2.17.1
From: SeongJae Park <[email protected]>
This commit adds documents for DAMON under
`Documentation/admin-guide/mm/damon/` and `Documentation/vm/damon/`.
Signed-off-by: SeongJae Park <[email protected]>
---
Documentation/admin-guide/mm/damon/guide.rst | 157 ++++++++++
Documentation/admin-guide/mm/damon/index.rst | 15 +
Documentation/admin-guide/mm/damon/plans.rst | 29 ++
Documentation/admin-guide/mm/damon/start.rst | 96 ++++++
Documentation/admin-guide/mm/damon/usage.rst | 302 +++++++++++++++++++
Documentation/admin-guide/mm/index.rst | 1 +
Documentation/vm/damon/api.rst | 20 ++
Documentation/vm/damon/design.rst | 166 ++++++++++
Documentation/vm/damon/eval.rst | 227 ++++++++++++++
Documentation/vm/damon/faq.rst | 58 ++++
Documentation/vm/damon/index.rst | 31 ++
Documentation/vm/index.rst | 1 +
12 files changed, 1103 insertions(+)
create mode 100644 Documentation/admin-guide/mm/damon/guide.rst
create mode 100644 Documentation/admin-guide/mm/damon/index.rst
create mode 100644 Documentation/admin-guide/mm/damon/plans.rst
create mode 100644 Documentation/admin-guide/mm/damon/start.rst
create mode 100644 Documentation/admin-guide/mm/damon/usage.rst
create mode 100644 Documentation/vm/damon/api.rst
create mode 100644 Documentation/vm/damon/design.rst
create mode 100644 Documentation/vm/damon/eval.rst
create mode 100644 Documentation/vm/damon/faq.rst
create mode 100644 Documentation/vm/damon/index.rst
diff --git a/Documentation/admin-guide/mm/damon/guide.rst b/Documentation/admin-guide/mm/damon/guide.rst
new file mode 100644
index 000000000000..c51fb843efaa
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/guide.rst
@@ -0,0 +1,157 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================
+Optimization Guide
+==================
+
+This document helps you estimating the amount of benefit that you could get
+from DAMON-based optimizations, and describes how you could achieve it. You
+are assumed to already read :doc:`start`.
+
+
+Check The Signs
+===============
+
+No optimization can provide same extent of benefit to every case. Therefore
+you should first guess how much improvements you could get using DAMON. If
+some of below conditions match your situation, you could consider using DAMON.
+
+- *Low IPC and High Cache Miss Ratios.* Low IPC means most of the CPU time is
+ spent waiting for the completion of time-consuming operations such as memory
+ access, while high cache miss ratios mean the caches don't help it well.
+ DAMON is not for cache level optimization, but DRAM level. However,
+ improving DRAM management will also help this case by reducing the memory
+ operation latency.
+- *Memory Over-commitment and Unknown Users.* If you are doing memory
+ overcommitment and you cannot control every user of your system, a memory
+ bank run could happen at any time. You can estimate when it will happen
+ based on DAMON's monitoring results and act earlier to avoid or deal better
+ with the crisis.
+- *Frequent Memory Pressure.* Frequent memory pressure means your system has
+ wrong configurations or memory hogs. DAMON will help you find the right
+ configuration and/or the criminals.
+- *Heterogeneous Memory System.* If your system is utilizing memory devices
+ that placed between DRAM and traditional hard disks, such as non-volatile
+ memory or fast SSDs, DAMON could help you utilizing the devices more
+ efficiently.
+
+
+Profile
+=======
+
+If you found some positive signals, you could start by profiling your workloads
+using DAMON. Find major workloads on your systems and analyze their data
+access pattern to find something wrong or can be improved. The DAMON user
+space tool (``damo``) will be useful for this.
+
+We recommend you to start from working set size distribution check using ``damo
+report wss``. If the distribution is ununiform or quite different from what
+you estimated, you could consider `Memory Configuration`_ optimization.
+
+Then, review the overall access pattern in heatmap form using ``damo report
+heats``. If it shows a simple pattern consists of a small number of memory
+regions having high contrast of access temperature, you could consider manual
+`Program Modification`_.
+
+If you still want to absorb more benefits, you should develop `Personalized
+DAMON Application`_ for your special case.
+
+You don't need to take only one approach among the above plans, but you could
+use multiple of the above approaches to maximize the benefit.
+
+
+Optimize
+========
+
+If the profiling result also says it's worth trying some optimization, you
+could consider below approaches. Note that some of the below approaches assume
+that your systems are configured with swap devices or other types of auxiliary
+memory so that you don't strictly required to accommodate the whole working set
+in the main memory. Most of the detailed optimization should be made on your
+concrete understanding of your memory devices.
+
+
+Memory Configuration
+--------------------
+
+No more no less, DRAM should be large enough to accommodate only important
+working sets, because DRAM is highly performance critical but expensive and
+heavily consumes the power. However, knowing the size of the real important
+working sets is difficult. As a consequence, people usually equips
+unnecessarily large or too small DRAM. Many problems stem from such wrong
+configurations.
+
+Using the working set size distribution report provided by ``damo report wss``,
+you can know the appropriate DRAM size for you. For example, roughly speaking,
+if you worry about only 95 percentile latency, you don't need to equip DRAM of
+a size larger than 95 percentile working set size.
+
+Let's see a real example. This `page
+<https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/guide.html#memory-configuration>`_
+shows the heatmap and the working set size distributions/changes of
+``freqmine`` workload in PARSEC3 benchmark suite. The working set size spikes
+up to 180 MiB, but keeps smaller than 50 MiB for more than 95% of the time.
+Even though you give only 50 MiB of memory space to the workload, it will work
+well for 95% of the time. Meanwhile, you can save the 130 MiB of memory space.
+
+
+Program Modification
+--------------------
+
+If the data access pattern heatmap plotted by ``damo report heats`` is quite
+simple so that you can understand how the things are going in the workload with
+your human eye, you could manually optimize the memory management.
+
+For example, suppose that the workload has two big memory object but only one
+object is frequently accessed while the other one is only occasionally
+accessed. Then, you could modify the program source code to keep the hot
+object in the main memory by invoking ``mlock()`` or ``madvise()`` with
+``MADV_WILLNEED``. Or, you could proactively evict the cold object using
+``madvise()`` with ``MADV_COLD`` or ``MADV_PAGEOUT``. Using both together
+would be also worthy.
+
+A research work [1]_ using the ``mlock()`` achieved up to 2.55x performance
+speedup.
+
+Let's see another realistic example access pattern for this kind of
+optimizations. This `page
+<https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/guide.html#program-modification>`_
+shows the visualized access patterns of streamcluster workload in PARSEC3
+benchmark suite. We can easily identify the 100 MiB sized hot object.
+
+
+Personalized DAMON Application
+------------------------------
+
+Above approaches will work well for many general cases, but would not enough
+for some special cases.
+
+If this is the case, it might be the time to forget the comfortable use of the
+user space tool and dive into the debugfs interface (refer to :doc:`usage` for
+the detail) of DAMON. Using the interface, you can control the DAMON more
+flexibly. Therefore, you can write your personalized DAMON application that
+controls the monitoring via the debugfs interface, analyzes the result, and
+applies complex optimizations itself. Using this, you can make more creative
+and wise optimizations.
+
+If you are a kernel space programmer, writing kernel space DAMON applications
+using the API (refer to the :doc:`/vm/damon/api` for more detail) would be an
+option.
+
+
+Reference Practices
+===================
+
+Referencing previously done successful practices could help you getting the
+sense for this kind of optimizations. There is an academic paper [1]_
+reporting the visualized access pattern and manual `Program
+Modification`_ results for a number of realistic workloads. You can also get
+the visualized access patterns [3]_ [4]_ [5]_ and automated DAMON-based memory
+operations results for other realistic workloads that collected with latest
+version of DAMON [2]_ .
+
+.. [1] https://dl.acm.org/doi/10.1145/3366626.3368125
+.. [2] https://damonitor.github.io/test/result/perf/latest/html/
+.. [3] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html
+.. [4] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html
+.. [5] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
diff --git a/Documentation/admin-guide/mm/damon/index.rst b/Documentation/admin-guide/mm/damon/index.rst
new file mode 100644
index 000000000000..0baae7a5402b
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/index.rst
@@ -0,0 +1,15 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+========================
+Monitoring Data Accesses
+========================
+
+:doc:`DAMON </vm/damon/index>` allows light-weight data access monitoring.
+Using this, users can analyze and optimize their systems.
+
+.. toctree::
+ :maxdepth: 2
+
+ start
+ guide
+ usage
diff --git a/Documentation/admin-guide/mm/damon/plans.rst b/Documentation/admin-guide/mm/damon/plans.rst
new file mode 100644
index 000000000000..e3aa5ab96c29
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/plans.rst
@@ -0,0 +1,29 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+============
+Future Plans
+============
+
+DAMON is still on its first stage. Below plans are still under development.
+
+
+Automate Data Access Monitoring-based Memory Operation Schemes Execution
+========================================================================
+
+The ultimate goal of DAMON is to be used as a building block for the data
+access pattern aware kernel memory management optimization. It will make
+system just works efficiently. However, some users having very special
+workloads will want to further do their own optimization. DAMON will automate
+most of the tasks for such manual optimizations in near future. Users will be
+required to only describe what kind of data access pattern-based operation
+schemes they want in a simple form.
+
+By applying a very simple scheme for THP promotion/demotion with a prototype
+implementation, DAMON reduced 60% of THP memory footprint overhead while
+preserving 50% of the THP performance benefit. The detailed results can be
+seen on an external web page [1]_.
+
+Several RFC patchsets for this plan are available [2]_.
+
+.. [1] https://damonitor.github.io/test/result/perf/latest/html/
+.. [2] https://lore.kernel.org/linux-mm/[email protected]/
diff --git a/Documentation/admin-guide/mm/damon/start.rst b/Documentation/admin-guide/mm/damon/start.rst
new file mode 100644
index 000000000000..deed2ea2321e
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/start.rst
@@ -0,0 +1,96 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+Getting Started
+===============
+
+This document briefly describes how you can use DAMON by demonstrating its
+default user space tool. Please note that this document describes only a part
+of its features for brevity. Please refer to :doc:`usage` for more details.
+
+
+TL; DR
+======
+
+Follow below 5 commands to monitor and visualize the access pattern of your
+workload. ::
+
+ $ git clone https://github.com/sjp38/linux -b damon/master
+ /* build the kernel with CONFIG_DAMON=y, install, reboot */
+ $ mount -t debugfs none /sys/kernel/debug/
+ $ cd linux/tools/damon
+ $ ./damo record $(pidof <your workload>)
+ $ ./damo report heats --heatmap access_pattern.png
+
+
+Prerequisites
+=============
+
+Kernel
+------
+
+You should first ensure your system is running on a kernel built with
+``CONFIG_DAMON=y``.
+
+
+User Space Tool
+---------------
+
+For the demonstration, we will use the default user space tool for DAMON,
+called DAMON Operator (DAMO). It is located at ``tools/damon/damo`` of the
+kernel source tree. For brevity, below examples assume you set ``$PATH`` to
+point it. It's not mandatory, though.
+
+Because DAMO is using the debugfs interface (refer to :doc:`usage` for the
+detail) of DAMON, you should ensure debugfs is mounted. Mount it manually as
+below::
+
+ # mount -t debugfs none /sys/kernel/debug/
+
+or append below line to your ``/etc/fstab`` file so that your system can
+automatically mount debugfs from next booting::
+
+ debugfs /sys/kernel/debug debugfs defaults 0 0
+
+
+Recording Data Access Patterns
+==============================
+
+Below commands record memory access pattern of a program and save the
+monitoring results in a file. ::
+
+ $ git clone https://github.com/sjp38/masim
+ $ cd masim; make; ./masim ./configs/zigzag.cfg &
+ $ sudo damo record -o damon.data $(pidof masim)
+
+The first two lines of the commands get an artificial memory access generator
+program and runs it in the background. It will repeatedly access two 100 MiB
+sized memory regions one by one. You can substitute this with your real
+workload. The last line asks ``damo`` to record the access pattern in
+``damon.data`` file.
+
+
+Visualizing Recorded Patterns
+=============================
+
+Below three commands visualize the recorded access patterns into three
+image files. ::
+
+ $ damo report heats --heatmap access_pattern_heatmap.png
+ $ damo report wss --range 0 101 1 --plot wss_dist.png
+ $ damo report wss --range 0 101 1 --sortby time --plot wss_chron_change.png
+
+- ``access_pattern_heatmap.png`` will show the data access pattern in a
+ heatmap, which shows when (x-axis) what memory region (y-axis) is how
+ frequently accessed (color).
+- ``wss_dist.png`` will show the distribution of the working set size.
+- ``wss_chron_change.png`` will show how the working set size has
+ chronologically changed.
+
+You can show the images in a web page [1]_ . Those made with other realistic
+workloads are also available [2]_ [3]_ [4]_.
+
+.. [1] https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/start.html#visualizing-recorded-patterns
+.. [2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html
+.. [3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html
+.. [4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst
new file mode 100644
index 000000000000..a6606d27a559
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/usage.rst
@@ -0,0 +1,302 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+Detailed Usages
+===============
+
+DAMON provides below three interfaces for different users.
+
+- *DAMON user space tool.*
+ This is for privileged people such as system administrators who want a
+ just-working human-friendly interface. Using this, users can use the DAMON’s
+ major features in a human-friendly way. It may not be highly tuned for
+ special cases, though. It supports only virtual address spaces monitoring.
+- *debugfs interface.*
+ This is for privileged user space programmers who want more optimized use of
+ DAMON. Using this, users can use DAMON’s major features by reading
+ from and writing to special debugfs files. Therefore, you can write and use
+ your personalized DAMON debugfs wrapper programs that reads/writes the
+ debugfs files instead of you. The DAMON user space tool is also a reference
+ implementation of such programs. It supports only virtual address spaces
+ monitoring.
+- *Kernel Space Programming Interface.*
+ This is for kernel space programmers. Using this, users can utilize every
+ feature of DAMON most flexibly and efficiently by writing kernel space
+ DAMON application programs for you. You can even extend DAMON for various
+ address spaces.
+
+This document does not describe the kernel space programming interface in
+detail. For that, please refer to the :doc:`/vm/damon/api`.
+
+
+DAMON User Space Tool
+=====================
+
+A reference implementation of the DAMON user space tools which provides a
+convenient user interface is in the kernel source tree. It is located at
+``tools/damon/damo`` of the tree.
+
+The tool provides a subcommands based interface. Every subcommand provides
+``-h`` option, which provides the minimal usage of it. Currently, the tool
+supports two subcommands, ``record`` and ``report``.
+
+Below example commands assume you set ``$PATH`` to point ``tools/damon/`` for
+brevity. It is not mandatory for use of ``damo``, though.
+
+
+Recording Data Access Pattern
+-----------------------------
+
+The ``record`` subcommand records the data access pattern of target workloads
+in a file (``./damon.data`` by default). You can specify the target with 1)
+the command for execution of the monitoring target process, or 2) pid of
+running target process. Below example shows a command target usage::
+
+ # cd <kernel>/tools/damon/
+ # damo record "sleep 5"
+
+The tool will execute ``sleep 5`` by itself and record the data access patterns
+of the process. Below example shows a pid target usage::
+
+ # sleep 5 &
+ # damo record `pidof sleep`
+
+The location of the recorded file can be explicitly set using ``-o`` option.
+You can further tune this by setting the monitoring attributes. To know about
+the monitoring attributes in detail, please refer to the
+:doc:`/vm/damon/design`.
+
+
+Analyzing Data Access Pattern
+-----------------------------
+
+The ``report`` subcommand reads a data access pattern record file (if not
+explicitly specified using ``-i`` option, reads ``./damon.data`` file by
+default) and generates human-readable reports. You can specify what type of
+report you want using a sub-subcommand to ``report`` subcommand. ``raw``,
+``heats``, and ``wss`` report types are supported for now.
+
+
+raw
+~~~
+
+``raw`` sub-subcommand simply transforms the binary record into a
+human-readable text. For example::
+
+ $ damo report raw
+ start_time: 193485829398
+ rel time: 0
+ nr_tasks: 1
+ target_id: 1348
+ nr_regions: 4
+ 560189609000-56018abce000( 22827008): 0
+ 7fbdff59a000-7fbdffaf1a00( 5601792): 0
+ 7fbdffaf1a00-7fbdffbb5000( 800256): 1
+ 7ffea0dc0000-7ffea0dfd000( 249856): 0
+
+ rel time: 100000731
+ nr_tasks: 1
+ target_id: 1348
+ nr_regions: 6
+ 560189609000-56018abce000( 22827008): 0
+ 7fbdff59a000-7fbdff8ce933( 3361075): 0
+ 7fbdff8ce933-7fbdffaf1a00( 2240717): 1
+ 7fbdffaf1a00-7fbdffb66d99( 480153): 0
+ 7fbdffb66d99-7fbdffbb5000( 320103): 1
+ 7ffea0dc0000-7ffea0dfd000( 249856): 0
+
+The first line shows the recording started timestamp (nanosecond). Records of
+data access patterns follows. Each record is separated by a blank line. Each
+record first specifies the recorded time (``rel time``) in relative to the
+start time, the number of monitored tasks in this record (``nr_tasks``).
+Recorded data access patterns of each task follow. Each data access pattern
+for each task shows the target's pid (``target_id``) and a number of monitored
+address regions in this access pattern (``nr_regions``) first. After that,
+each line shows the start/end address, size, and the number of observed
+accesses of each region.
+
+
+heats
+~~~~~
+
+The ``raw`` output is very detailed but hard to manually read. ``heats``
+sub-subcommand plots the data in 3-dimensional form, which represents the time
+in x-axis, address of regions in y-axis, and the access frequency in z-axis.
+Users can set the resolution of the map (``--tres`` and ``--ares``) and
+start/end point of each axis (``--tmin``, ``--tmax``, ``--amin``, and
+``--amax``) via optional arguments. For example::
+
+ $ damo report heats --tres 3 --ares 3
+ 0 0 0.0
+ 0 7609002 0.0
+ 0 15218004 0.0
+ 66112620851 0 0.0
+ 66112620851 7609002 0.0
+ 66112620851 15218004 0.0
+ 132225241702 0 0.0
+ 132225241702 7609002 0.0
+ 132225241702 15218004 0.0
+
+This command shows a recorded access pattern in heatmap of 3x3 resolution.
+Therefore it shows 9 data points in total. Each line shows each of the data
+points. The three numbers in each line represent time in nanosecond, address,
+and the observed access frequency.
+
+Users will be able to convert this text output into a heatmap image (represents
+z-axis values with colors) or other 3D representations using various tools such
+as 'gnuplot'. For more convenience, ``heats`` sub-subcommand provides the
+'gnuplot' based heatmap image creation. For this, you can use ``--heatmap``
+option. Also, note that because it uses 'gnuplot' internally, it will fail if
+'gnuplot' is not installed on your system. For example::
+
+ $ ./damo report heats --heatmap heatmap.png
+
+Creates the heatmap image in ``heatmap.png`` file. It supports ``pdf``,
+``png``, ``jpeg``, and ``svg``.
+
+If the target address space is virtual memory address space and you plot the
+entire address space, the huge unmapped regions will make the picture looks
+only black. Therefore you should do proper zoom in / zoom out using the
+resolution and axis boundary-setting arguments. To make this effort minimal,
+you can use ``--guide`` option as below::
+
+ $ ./damo report heats --guide
+ target_id:1348
+ time: 193485829398-198337863555 (4852034157)
+ region 0: 00000094564599762944-00000094564622589952 (22827008)
+ region 1: 00000140454009610240-00000140454016012288 (6402048)
+ region 2: 00000140731597193216-00000140731597443072 (249856)
+
+The output shows unions of monitored regions (start and end addresses in byte)
+and the union of monitored time duration (start and end time in nanoseconds) of
+each target task. Therefore, it would be wise to plot the data points in each
+union. If no axis boundary option is given, it will automatically find the
+biggest union in ``--guide`` output and set the boundary in it.
+
+
+wss
+~~~
+
+The ``wss`` type extracts the distribution and chronological working set size
+changes from the records. For example::
+
+ $ ./damo report wss
+ # <percentile> <wss>
+ # target_id 1348
+ # avr: 66228
+ 0 0
+ 25 0
+ 50 0
+ 75 0
+ 100 1920615
+
+Without any option, it shows the distribution of the working set sizes as
+above. It shows 0th, 25th, 50th, 75th, and 100th percentile and the average of
+the measured working set sizes in the access pattern records. In this case,
+the working set size was zero for 75th percentile but 1,920,615 bytes in max
+and 66,228 bytes on average.
+
+By setting the sort key of the percentile using '--sortby', you can show how
+the working set size has chronologically changed. For example::
+
+ $ ./damo report wss --sortby time
+ # <percentile> <wss>
+ # target_id 1348
+ # avr: 66228
+ 0 0
+ 25 0
+ 50 0
+ 75 0
+ 100 0
+
+The average is still 66,228. And, because the access was spiked in very short
+duration and this command plots only 4 data points, we cannot show when the
+access spikes made. Users can specify the resolution of the distribution
+(``--range``). By giving more fine resolution, the short duration spikes could
+be found.
+
+Similar to that of ``heats --heatmap``, it also supports 'gnuplot' based simple
+visualization of the distribution via ``--plot`` option.
+
+
+debugfs Interface
+=================
+
+DAMON exports four files, ``attrs``, ``target_ids``, ``record``, and
+``monitor_on`` under its debugfs directory, ``<debugfs>/damon/``.
+
+
+Attributes
+----------
+
+Users can get and set the ``sampling interval``, ``aggregation interval``,
+``regions update interval``, and min/max number of monitoring target regions by
+reading from and writing to the ``attrs`` file. To know about the monitoring
+attributes in detail, please refer to the :doc:`/vm/damon/design`. For
+example, below commands set those values to 5 ms, 100 ms, 1,000 ms, 10 and
+1000, and then check it again::
+
+ # cd <debugfs>/damon
+ # echo 5000 100000 1000000 10 1000 > attrs
+ # cat attrs
+ 5000 100000 1000000 10 1000
+
+
+Target IDs
+----------
+
+Some types of address spaces supports multiple monitoring target. For example,
+the virtual memory address spaces monitoring can have multiple processes as the
+monitoring targets. Users can set the targets by writing relevant id values of
+the targets to, and get the ids of the current targets by reading from the
+``target_ids`` file. In case of the virtual address spaces monitoring, the
+values should be pids of the monitoring target processes. For example, below
+commands set processes having pids 42 and 4242 as the monitoring targets and
+check it again::
+
+ # cd <debugfs>/damon
+ # echo 42 4242 > target_ids
+ # cat target_ids
+ 42 4242
+
+Note that setting the target ids doesn't start the monitoring.
+
+
+Record
+------
+
+This debugfs file allows you to record monitored access patterns in a regular
+binary file. The recorded results are first written in an in-memory buffer and
+flushed to a file in batch. Users can get and set the size of the buffer and
+the path to the result file by reading from and writing to the ``record`` file.
+For example, below commands set the buffer to be 4 KiB and the result to be
+saved in ``/damon.data``. ::
+
+ # cd <debugfs>/damon
+ # echo "4096 /damon.data" > record
+ # cat record
+ 4096 /damon.data
+
+The recording can be disabled by setting the buffer size zero.
+
+
+Turning On/Off
+--------------
+
+Setting the files as described above doesn't incur effect unless you explicitly
+start the monitoring. You can start, stop, and check the current status of the
+monitoring by writing to and reading from the ``monitor_on`` file. Writing
+``on`` to the file starts the monitoring of the targets with the attributes.
+Writing ``off`` to the file stops those. DAMON also stops if every target
+process is terminated. Below example commands turn on, off, and check the
+status of DAMON::
+
+ # cd <debugfs>/damon
+ # echo on > monitor_on
+ # echo off > monitor_on
+ # cat monitor_on
+ off
+
+Please note that you cannot write to the above-mentioned debugfs files while
+the monitoring is turned on. If you write to the files while DAMON is running,
+an error code such as ``-EBUSY`` will be returned.
diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst
index cd727cfc1b04..32c27fbf1913 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -27,6 +27,7 @@ the Linux memory management.
concepts
cma_debugfs
+ damon/index
hugetlbpage
idle_page_tracking
ksm
diff --git a/Documentation/vm/damon/api.rst b/Documentation/vm/damon/api.rst
new file mode 100644
index 000000000000..08f34df45523
--- /dev/null
+++ b/Documentation/vm/damon/api.rst
@@ -0,0 +1,20 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=============
+API Reference
+=============
+
+Kernel space programs can use every feature of DAMON using below APIs. All you
+need to do is including ``damon.h``, which is located in ``include/linux/`` of
+the source tree.
+
+Structures
+==========
+
+.. kernel-doc:: include/linux/damon.h
+
+
+Functions
+=========
+
+.. kernel-doc:: mm/damon/core.c
diff --git a/Documentation/vm/damon/design.rst b/Documentation/vm/damon/design.rst
new file mode 100644
index 000000000000..727d72093f8f
--- /dev/null
+++ b/Documentation/vm/damon/design.rst
@@ -0,0 +1,166 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======
+Design
+======
+
+Configurable Layers
+===================
+
+DAMON provides data access monitoring functionality while making the accuracy
+and the overhead controllable. The fundamental access monitorings require
+primitives that dependent on and optimized for the target address space. On
+the other hand, the accuracy and overhead tradeoff mechanism, which is the core
+of DAMON, is in the pure logic space. DAMON separates the two parts in
+different layers and defines its interface to allow various low level
+primitives implementations configurable with the core logic.
+
+Due to this separated design and the configurable interface, users can extend
+DAMON for any address space by configuring the core logics with appropriate low
+level primitive implementations. If appropriate one is not provided, users can
+implement the primitives on their own.
+
+For example, physical memory, virtual memory, swap space, those for specific
+processes, NUMA nodes, files, and backing memory devices would be supportable.
+Also, if some architectures or devices support special optimized access check
+primitives, those will be easily configurable.
+
+
+Reference Implementations of Address Space Specific Primitives
+==============================================================
+
+The low level primitives for the fundamental access monitoring are defined in
+two parts:
+
+1. Identification of the monitoring target address range for the address space.
+2. Access check of specific address range in the target space.
+
+DAMON currently provides the implementation of the primitives for only the
+virtual address spaces. Below two subsections describe how it works.
+
+
+PTE Accessed-bit Based Access Check
+-----------------------------------
+
+The implementation for the virtual address space uses PTE Accessed-bit for
+basic access checks. It finds the relevant PTE Accessed bit from the address
+by walking the page table for the target task of the address. In this way, the
+implementation finds and clears the bit for next sampling target address and
+checks whether the bit set again after one sampling period. This could disturb
+other kernel subsystems using the Accessed bits, namely Idle page tracking and
+the reclaim logic. To avoid such disturbances, DAMON makes it mutually
+exclusive with Idle page tracking and uses ``PG_idle`` and ``PG_young`` page
+flags to solve the conflict with the reclaim logic, as Idle page tracking does.
+
+
+VMA-based Target Address Range Construction
+-------------------------------------------
+
+Only small parts in the super-huge virtual address space of the processes are
+mapped to the physical memory and accessed. Thus, tracking the unmapped
+address regions is just wasteful. However, because DAMON can deal with some
+level of noise using the adaptive regions adjustment mechanism, tracking every
+mapping is not strictly required but could even incur a high overhead in some
+cases. That said, too huge unmapped areas inside the monitoring target should
+be removed to not take the time for the adaptive mechanism.
+
+For the reason, this implementation converts the complex mappings to three
+distinct regions that cover every mapped area of the address space. The two
+gaps between the three regions are the two biggest unmapped areas in the given
+address space. The two biggest unmapped areas would be the gap between the
+heap and the uppermost mmap()-ed region, and the gap between the lowermost
+mmap()-ed region and the stack in most of the cases. Because these gaps are
+exceptionally huge in usual address spaces, excluding these will be sufficient
+to make a reasonable trade-off. Below shows this in detail::
+
+ <heap>
+ <BIG UNMAPPED REGION 1>
+ <uppermost mmap()-ed region>
+ (small mmap()-ed regions and munmap()-ed regions)
+ <lowermost mmap()-ed region>
+ <BIG UNMAPPED REGION 2>
+ <stack>
+
+
+Address Space Independent Core Mechanisms
+=========================================
+
+Below four sections describe each of the DAMON core mechanisms and the five
+monitoring attributes, ``sampling interval``, ``aggregation interval``,
+``regions update interval``, ``minimum number of regions``, and ``maximum
+number of regions``.
+
+
+Access Frequency Monitoring
+---------------------------
+
+The output of DAMON says what pages are how frequently accessed for a given
+duration. The resolution of the access frequency is controlled by setting
+``sampling interval`` and ``aggregation interval``. In detail, DAMON checks
+access to each page per ``sampling interval`` and aggregates the results. In
+other words, counts the number of the accesses to each page. After each
+``aggregation interval`` passes, DAMON calls callback functions that previously
+registered by users so that users can read the aggregated results and then
+clears the results. This can be described in below simple pseudo-code::
+
+ while monitoring_on:
+ for page in monitoring_target:
+ if accessed(page):
+ nr_accesses[page] += 1
+ if time() % aggregation_interval == 0:
+ for callback in user_registered_callbacks:
+ callback(monitoring_target, nr_accesses)
+ for page in monitoring_target:
+ nr_accesses[page] = 0
+ sleep(sampling interval)
+
+The monitoring overhead of this mechanism will arbitrarily increase as the
+size of the target workload grows.
+
+
+Region Based Sampling
+---------------------
+
+To avoid the unbounded increase of the overhead, DAMON groups adjacent pages
+that assumed to have the same access frequencies into a region. As long as the
+assumption (pages in a region have the same access frequencies) is kept, only
+one page in the region is required to be checked. Thus, for each ``sampling
+interval``, DAMON randomly picks one page in each region, waits for one
+``sampling interval``, checks whether the page is accessed meanwhile, and
+increases the access frequency of the region if so. Therefore, the monitoring
+overhead is controllable by setting the number of regions. DAMON allows users
+to set the minimum and the maximum number of regions for the trade-off.
+
+This scheme, however, cannot preserve the quality of the output if the
+assumption is not guaranteed.
+
+
+Adaptive Regions Adjustment
+---------------------------
+
+Even somehow the initial monitoring target regions are well constructed to
+fulfill the assumption (pages in same region have similar access frequencies),
+the data access pattern can be dynamically changed. This will result in low
+monitoring quality. To keep the assumption as much as possible, DAMON
+adaptively merges and splits each region based on their access frequency.
+
+For each ``aggregation interval``, it compares the access frequencies of
+adjacent regions and merges those if the frequency difference is small. Then,
+after it reports and clears the aggregated access frequency of each region, it
+splits each region into two or three regions if the total number of regions
+will not exceed the user-specified maximum number of regions after the split.
+
+In this way, DAMON provides its best-effort quality and minimal overhead while
+keeping the bounds users set for their trade-off.
+
+
+Dynamic Target Space Updates Handling
+-------------------------------------
+
+The monitoring target address range could dynamically changed. For example,
+virtual memory could be dynamically mapped and unmapped. Physical memory could
+be hot-plugged.
+
+As the changes could be quite frequent in some cases, DAMON checks the dynamic
+memory mapping changes and applies it to the abstracted target area only for
+each of a user-specified time interval (``regions update interval``).
diff --git a/Documentation/vm/damon/eval.rst b/Documentation/vm/damon/eval.rst
new file mode 100644
index 000000000000..ca97e6334aa7
--- /dev/null
+++ b/Documentation/vm/damon/eval.rst
@@ -0,0 +1,227 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========
+Evaluation
+==========
+
+DAMON is lightweight. It increases system memory usage by 0.42% and slows
+target workloads down by 0.39%.
+
+DAMON is accurate and useful for memory management optimizations. An
+experimental DAMON-based operation scheme for THP, namely 'ethp', removes
+81.45% of THP memory overheads while preserving 50.09% of THP speedup. Another
+experimental DAMON-based 'proactive reclamation' implementation, namely 'prcl',
+reduces 91.45% of residential sets and 22.91% of system memory footprint while
+incurring only 2.43% runtime overhead in the best case (parsec3/freqmine).
+
+
+Setup
+=====
+
+On QEMU/KVM based virtual machines utilizing 130GB of RAM and 36 vCPUs hosted
+by AWS EC2 i3.metal instances that running a kernel that v21 DAMON patchset is
+applied, I measure runtime and consumed system memory while running various
+realistic workloads with several configurations. From each of PARSEC3 [3]_ and
+SPLASH-2X [4]_ benchmark suites I pick 12 workloads, so I use 24 workloads in
+total. I use another wrapper scripts [5]_ for convenient setup and run of the
+workloads.
+
+
+Measurement
+-----------
+
+For the measurement of the amount of consumed memory in system global scope, I
+drop caches before starting each of the workloads and monitor 'MemFree' in the
+'/proc/meminfo' file. To make results more stable, I repeat the runs 5 times
+and average results.
+
+
+Configurations
+--------------
+
+The configurations I use are as below.
+
+- orig: Linux v5.10 with 'madvise' THP policy
+- rec: 'orig' plus DAMON running with virtual memory access recording
+- prec: 'orig' plus DAMON running with physical memory access recording
+- thp: same with 'orig', but use 'always' THP policy
+- ethp: 'orig' plus a DAMON operation scheme, 'efficient THP'
+- prcl: 'orig' plus a DAMON operation scheme, 'proactive reclaim [6]_'
+
+I use 'rec' for measurement of DAMON overheads to target workloads and system
+memory. 'prec' is for physical memory monitroing and recording. It monitors
+17GB sized 'System RAM' region. The remaining configs including 'thp', 'ethp',
+and 'prcl' are for measurement of DAMON monitoring accuracy.
+
+'ethp' and 'prcl' are simple DAMON-based operation schemes developed for
+proof of concepts of DAMON. 'ethp' reduces memory space waste of THP by using
+DAMON for the decision of promotions and demotion for huge pages, while 'prcl'
+is as similar as the original work. Those are implemented as below::
+
+ # format: <min/max size> <min/max frequency (0-100)> <min/max age> <action>
+ # ethp: Use huge pages if a region shows >=5% access rate, use regular
+ # pages if a region >=2MB shows 0 access rate for >=7 seconds
+ min max 5 max min max hugepage
+ 2M max min min 7s max nohugepage
+
+ # prcl: If a region >=4KB shows 0 access rate for >=5 seconds, page out.
+ 4K max 0 0 5s max pageout
+
+Note that both 'ethp' and 'prcl' are designed with my only straightforward
+intuition because those are for only proof of concepts and monitoring accuracy
+of DAMON. In other words, those are not for production. For production use,
+those should be more tuned.
+
+The evaluation is done using the tests package for DAMON, ``damon-tests`` [7]_.
+Using it, you can do the evaluation and generate a report on your own.
+
+.. [1] "Redis latency problems troubleshooting", https://redis.io/topics/latency
+.. [2] "Disable Transparent Huge Pages (THP)",
+ https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
+.. [3] "The PARSEC Becnhmark Suite", https://parsec.cs.princeton.edu/index.htm
+.. [4] "SPLASH-2x", https://parsec.cs.princeton.edu/parsec3-doc.htm#splash2x
+.. [5] "parsec3_on_ubuntu", https://github.com/sjp38/parsec3_on_ubuntu
+.. [6] "Proactively reclaiming idle memory", https://lwn.net/Articles/787611/
+.. [7] "damon-tests", https://github.com/awslabs/damon-tests
+
+
+Results
+=======
+
+Below two tables show the measurement results. The runtimes are in seconds
+while the memory usages are in KiB. Each configuration except 'orig' shows
+its overhead relative to 'orig' in percent within parenthesizes.::
+
+ runtime orig rec (overhead) prec (overhead) thp (overhead) ethp (overhead) prcl (overhead)
+ parsec3/blackscholes 138.247 139.131 (0.64) 138.872 (0.45) 138.436 (0.14) 138.599 (0.25) 151.104 (9.30)
+ parsec3/bodytrack 124.338 124.450 (0.09) 124.624 (0.23) 124.357 (0.02) 124.705 (0.29) 125.329 (0.80)
+ parsec3/canneal 211.054 216.642 (2.65) 213.773 (1.29) 176.039 (-16.59) 214.460 (1.61) 249.492 (18.21)
+ parsec3/dedup 18.452 18.218 (-1.27) 18.334 (-0.64) 18.074 (-2.05) 18.315 (-0.74) 20.489 (11.04)
+ parsec3/facesim 347.473 352.724 (1.51) 345.951 (-0.44) 340.480 (-2.01) 344.158 (-0.95) 371.561 (6.93)
+ parsec3/fluidanimate 339.895 337.531 (-0.70) 335.378 (-1.33) 326.410 (-3.97) 333.322 (-1.93) 332.785 (-2.09)
+ parsec3/freqmine 436.827 437.962 (0.26) 439.345 (0.58) 436.844 (0.00) 438.301 (0.34) 447.430 (2.43)
+ parsec3/raytrace 185.539 183.376 (-1.17) 185.962 (0.23) 186.311 (0.42) 184.981 (-0.30) 207.715 (11.95)
+ parsec3/streamcluster 682.926 686.849 (0.57) 677.420 (-0.81) 599.544 (-12.21) 615.506 (-9.87) 789.596 (15.62)
+ parsec3/swaptions 219.616 221.386 (0.81) 221.320 (0.78) 220.269 (0.30) 221.426 (0.82) -100.000 (0.00)
+ parsec3/vips 88.397 88.504 (0.12) 87.550 (-0.96) 87.801 (-0.67) 87.638 (-0.86) 89.135 (0.84)
+ parsec3/x264 113.634 114.143 (0.45) 116.506 (2.53) 112.728 (-0.80) 116.572 (2.59) 114.607 (0.86)
+ splash2x/barnes 130.160 130.475 (0.24) 130.006 (-0.12) 119.679 (-8.05) 128.869 (-0.99) 173.767 (33.50)
+ splash2x/fft 61.243 60.419 (-1.35) 60.144 (-1.79) 46.930 (-23.37) 58.679 (-4.19) 94.651 (54.55)
+ splash2x/lu_cb 132.438 132.733 (0.22) 132.746 (0.23) 131.756 (-0.52) 132.492 (0.04) 146.579 (10.68)
+ splash2x/lu_ncb 151.133 150.656 (-0.32) 151.187 (0.04) 150.106 (-0.68) 149.088 (-1.35) 156.120 (3.30)
+ splash2x/ocean_cp 87.010 88.161 (1.32) 90.317 (3.80) 77.344 (-11.11) 77.739 (-10.65) 113.273 (30.18)
+ splash2x/ocean_ncp 161.819 160.428 (-0.86) 161.508 (-0.19) 117.250 (-27.54) 141.303 (-12.68) 279.021 (72.43)
+ splash2x/radiosity 144.159 142.662 (-1.04) 145.874 (1.19) 141.937 (-1.54) 142.184 (-1.37) 151.460 (5.06)
+ splash2x/radix 51.341 51.156 (-0.36) 51.601 (0.51) 46.678 (-9.08) 49.119 (-4.33) 82.058 (59.83)
+ splash2x/raytrace 133.543 134.201 (0.49) 134.022 (0.36) 132.010 (-1.15) 133.065 (-0.36) 141.626 (6.05)
+ splash2x/volrend 120.229 120.489 (0.22) 121.690 (1.22) 119.702 (-0.44) 119.693 (-0.45) 122.247 (1.68)
+ splash2x/water_nsquared 371.382 375.238 (1.04) 373.726 (0.63) 355.410 (-4.30) 358.243 (-3.54) 403.058 (8.53)
+ splash2x/water_spatial 133.738 134.831 (0.82) 133.865 (0.10) 133.270 (-0.35) 133.320 (-0.31) 152.743 (14.21)
+ total 4584.600 4602.380 (0.39) 4591.740 (0.16) 4339.370 (-5.35) 4461.770 (-2.68) 4915.870 (7.23)
+
+
+ memused.avg orig rec (overhead) prec (overhead) thp (overhead) ethp (overhead) prcl (overhead)
+ parsec3/blackscholes 1822419.200 1832932.800 (0.58) 1825942.600 (0.19) 1817011.600 (-0.30) 1830445.600 (0.44) 1595311.600 (-12.46)
+ parsec3/bodytrack 1424439.600 1437080.200 (0.89) 1438747.200 (1.00) 1423658.600 (-0.05) 1434771.600 (0.73) 1437144.200 (0.89)
+ parsec3/canneal 1036933.000 1054711.800 (1.71) 1050022.200 (1.26) 1032368.400 (-0.44) 1052744.400 (1.52) 1049121.200 (1.18)
+ parsec3/dedup 2500773.600 2502254.800 (0.06) 2467656.000 (-1.32) 2511153.400 (0.42) 2495594.600 (-0.21) 2488489.200 (-0.49)
+ parsec3/facesim 535653.600 550504.000 (2.77) 547305.400 (2.18) 542355.200 (1.25) 552392.400 (3.12) 484499.000 (-9.55)
+ parsec3/fluidanimate 572288.600 585018.400 (2.22) 582106.200 (1.72) 571557.400 (-0.13) 583349.400 (1.93) 493663.400 (-13.74)
+ parsec3/freqmine 982803.000 997657.400 (1.51) 995492.200 (1.29) 986962.000 (0.42) 998352.400 (1.58) 757675.800 (-22.91)
+ parsec3/raytrace 1742834.000 1754262.200 (0.66) 1747630.800 (0.28) 1731301.600 (-0.66) 1749506.400 (0.38) 1543049.400 (-11.46)
+ parsec3/streamcluster 117851.800 158437.400 (34.44) 158582.400 (34.56) 122982.600 (4.35) 135280.200 (14.79) 136526.600 (15.85)
+ parsec3/swaptions 14375.800 28709.600 (99.71) 28302.000 (96.87) 13821.400 (-3.86) 25697.800 (78.76) -100.000 (0.00)
+ parsec3/vips 2982188.400 2998594.600 (0.55) 3004458.800 (0.75) 2981225.200 (-0.03) 2997897.400 (0.53) 2979860.000 (-0.08)
+ parsec3/x264 3241201.800 3250602.600 (0.29) 3257842.600 (0.51) 3238675.800 (-0.08) 3254314.200 (0.40) 3243305.667 (0.06)
+ splash2x/barnes 1202953.000 1212273.400 (0.77) 1199432.200 (-0.29) 1214065.600 (0.92) 1218764.400 (1.31) 881206.000 (-26.75)
+ splash2x/fft 9729496.200 9631956.200 (-1.00) 9282596.600 (-4.59) 9892176.200 (1.67) 9632687.800 (-0.99) 10320735.333 (6.08)
+ splash2x/lu_cb 512464.200 523658.200 (2.18) 515659.200 (0.62) 513609.000 (0.22) 520062.000 (1.48) 338391.667 (-33.97)
+ splash2x/lu_ncb 512790.400 528954.400 (3.15) 521128.600 (1.63) 513166.000 (0.07) 523937.800 (2.17) 426409.333 (-16.85)
+ splash2x/ocean_cp 3342031.600 3326082.400 (-0.48) 3258501.400 (-2.50) 3367646.400 (0.77) 3314408.400 (-0.83) 3181677.000 (-4.80)
+ splash2x/ocean_ncp 3904158.200 3922279.200 (0.46) 3870676.800 (-0.86) 7071312.600 (81.12) 4513390.200 (15.60) 3517213.000 (-9.91)
+ splash2x/radiosity 1460571.200 1463947.200 (0.23) 1454906.200 (-0.39) 1470355.800 (0.67) 1465063.200 (0.31) 450619.333 (-69.15)
+ splash2x/radix 2379050.200 2377324.000 (-0.07) 2270805.200 (-4.55) 2477275.200 (4.13) 2313398.800 (-2.76) 2433462.333 (2.29)
+ splash2x/raytrace 42587.000 55138.400 (29.47) 55933.200 (31.34) 49202.200 (15.53) 59114.400 (38.81) 50805.000 (19.30)
+ splash2x/volrend 149927.000 163164.400 (8.83) 161644.400 (7.82) 149249.000 (-0.45) 160589.600 (7.11) 159004.000 (6.05)
+ splash2x/water_nsquared 39653.400 54180.600 (36.64) 53137.800 (34.01) 42475.000 (7.12) 52911.800 (33.44) 47500.333 (19.79)
+ splash2x/water_spatial 669766.600 681525.600 (1.76) 674610.800 (0.72) 670925.800 (0.17) 679559.000 (1.46) 405725.667 (-39.42)
+ total 40919400.000 41091400.000 (0.42) 40423000.000 (-1.21) 44404600.000 (8.52) 41564259.000 (1.58) 38421300.000 (-6.10)
+
+
+DAMON Overheads
+---------------
+
+In total, DAMON virtual memory access recording feature ('rec') incurs 0.39%
+runtime overhead and 0.42% memory space overhead. Even though the size of the
+monitoring target region becomes much larger with the physical memory access
+recording ('prec'), it still shows only modest amount of overhead (0.16% for
+runtime and -1.21% for memory footprint).
+
+For a convenient test run of 'rec' and 'prec', I use a Python wrapper. The
+wrapper constantly consumes about 10-15MB of memory. This becomes a high
+memory overhead if the target workload has a small memory footprint.
+Nonetheless, the overheads are not from DAMON, but from the wrapper, and thus
+should be ignored. This fake memory overhead continues in 'ethp' and 'prcl',
+as those configurations are also using the Python wrapper.
+
+
+Efficient THP
+-------------
+
+THP 'always' enabled policy achieves 5.35% speedup but incurs 8.52% memory
+overhead. It achieves 27.54% speedup in the best case, but 81.72% memory
+overhead in the worst case. Interestingly, both the best and worst-case are
+with 'splash2x/ocean_ncp').
+
+The 2-lines implementation of data access monitoring based THP version ('ethp')
+shows 2.68% speedup and 1.58% memory overhead. In other words, 'ethp' removes
+81.45% of THP memory waste while preserving 50.09% of THP speedup in total. In
+the case of the 'splash2x/ocean_ncp', 'ethp' removes 80.76% of THP memory waste
+while preserving 46.04% of THP speedup.
+
+
+Proactive Reclamation
+---------------------
+
+As similar to the original work, I use 4G 'zram' swap device for this
+configuration.
+
+In total, our 1 line implementation of Proactive Reclamation, 'prcl', incurred
+7.23% runtime overhead in total while achieving 6.10% system memory footprint
+reduction.
+
+Nonetheless, as the memory usage is calculated with 'MemFree' in
+'/proc/meminfo', it contains the SwapCached pages. As the swapcached pages can
+be easily evicted, I also measured the residential set size of the workloads::
+
+ rss.avg orig rec (overhead) prec (overhead) thp (overhead) ethp (overhead) prcl (overhead)
+ parsec3/blackscholes 585288.600 586175.800 (0.15) 586433.800 (0.20) 587028.600 (0.30) 587996.000 (0.46) 240808.600 (-58.86)
+ parsec3/bodytrack 32139.000 32312.400 (0.54) 32201.800 (0.20) 32357.000 (0.68) 32263.000 (0.39) 18371.000 (-42.84)
+ parsec3/canneal 843125.000 842998.800 (-0.01) 842991.000 (-0.02) 837536.400 (-0.66) 843580.600 (0.05) 825739.000 (-2.06)
+ parsec3/dedup 1187272.400 1175883.400 (-0.96) 1183341.800 (-0.33) 1192656.600 (0.45) 1178204.600 (-0.76) 582322.000 (-50.95)
+ parsec3/facesim 311757.600 311792.200 (0.01) 311751.400 (-0.00) 317679.400 (1.90) 315929.200 (1.34) 187274.800 (-39.93)
+ parsec3/fluidanimate 531844.800 531840.800 (-0.00) 531816.800 (-0.01) 532855.200 (0.19) 532576.400 (0.14) 439993.400 (-17.27)
+ parsec3/freqmine 552634.600 552707.800 (0.01) 552549.600 (-0.02) 555529.400 (0.52) 554548.200 (0.35) 47231.400 (-91.45)
+ parsec3/raytrace 887301.000 883878.400 (-0.39) 884147.800 (-0.36) 874717.000 (-1.42) 881240.200 (-0.68) 264899.000 (-70.15)
+ parsec3/streamcluster 110901.000 110899.200 (-0.00) 110906.200 (0.00) 115357.800 (4.02) 115521.800 (4.17) 109695.400 (-1.09)
+ parsec3/swaptions 5697.800 5682.600 (-0.27) 5704.400 (0.12) 5684.000 (-0.24) 5668.600 (-0.51) -100.000 (0.00)
+ parsec3/vips 32083.400 31877.000 (-0.64) 31873.800 (-0.65) 33041.200 (2.99) 33781.600 (5.29) 28844.667 (-10.09)
+ parsec3/x264 81776.600 81616.600 (-0.20) 81822.800 (0.06) 84827.400 (3.73) 83490.400 (2.10) 81161.333 (-0.75)
+ splash2x/barnes 1219285.200 1218478.600 (-0.07) 1218261.800 (-0.08) 1227469.800 (0.67) 1222605.400 (0.27) 460175.000 (-62.26)
+ splash2x/fft 10080559.600 10000486.200 (-0.79) 9996101.600 (-0.84) 10296965.200 (2.15) 9974327.200 (-1.05) 6932814.000 (-31.23)
+ splash2x/lu_cb 511985.800 511815.600 (-0.03) 511759.600 (-0.04) 511275.800 (-0.14) 511932.400 (-0.01) 319837.000 (-37.53)
+ splash2x/lu_ncb 511416.400 511389.800 (-0.01) 511257.800 (-0.03) 511574.800 (0.03) 511356.400 (-0.01) 412134.333 (-19.41)
+ splash2x/ocean_cp 3424155.800 3421099.600 (-0.09) 3415628.600 (-0.25) 3443500.000 (0.56) 3415558.200 (-0.25) 2436061.333 (-28.86)
+ splash2x/ocean_ncp 3939855.400 3934175.600 (-0.14) 3938673.800 (-0.03) 7177367.200 (82.17) 4581698.000 (16.29) 2391616.000 (-39.30)
+ splash2x/radiosity 1471925.400 1418593.800 (-3.62) 1474347.000 (0.16) 1485447.800 (0.92) 1475442.800 (0.24) 144195.333 (-90.20)
+ splash2x/radix 2465408.000 2484122.000 (0.76) 2449926.000 (-0.63) 2562083.200 (3.92) 2403580.400 (-2.51) 1539977.333 (-37.54)
+ splash2x/raytrace 23279.200 23288.800 (0.04) 23319.200 (0.17) 29137.000 (25.16) 26747.000 (14.90) 13287.667 (-42.92)
+ splash2x/volrend 44203.600 44115.000 (-0.20) 43493.000 (-1.61) 45079.000 (1.98) 45301.000 (2.48) 30139.333 (-31.82)
+ splash2x/water_nsquared 29424.000 29413.600 (-0.04) 29425.600 (0.01) 30163.800 (2.51) 30527.400 (3.75) 22633.667 (-23.08)
+ splash2x/water_spatial 663586.800 664276.200 (0.10) 664012.800 (0.06) 664078.800 (0.07) 663830.800 (0.04) 299712.667 (-54.83)
+ total 29547000.000 29408900.000 (-0.47) 29431800.000 (-0.39) 33153500.000 (12.21) 30027700.000 (1.63) 17828900.000 (-39.66)
+
+In total, 39.66% of residential sets were reduced.
+
+With parsec3/freqmine, 'prcl' reduced 91.45% of residential sets and 22.91% of
+system memory usage while incurring only 2.43% runtime overhead.
diff --git a/Documentation/vm/damon/faq.rst b/Documentation/vm/damon/faq.rst
new file mode 100644
index 000000000000..088128bbf22b
--- /dev/null
+++ b/Documentation/vm/damon/faq.rst
@@ -0,0 +1,58 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========================
+Frequently Asked Questions
+==========================
+
+Why a new subsystem, instead of extending perf or other user space tools?
+=========================================================================
+
+First, because it needs to be lightweight as much as possible so that it can be
+used online, any unnecessary overhead such as kernel - user space context
+switching cost should be avoided. Second, DAMON aims to be used by other
+programs including the kernel. Therefore, having a dependency on specific
+tools like perf is not desirable. These are the two biggest reasons why DAMON
+is implemented in the kernel space.
+
+
+Can 'idle pages tracking' or 'perf mem' substitute DAMON?
+=========================================================
+
+Idle page tracking is a low level primitive for access check of the physical
+address space. 'perf mem' is similar, though it can use sampling to minimize
+the overhead. On the other hand, DAMON is a higher-level framework for the
+monitoring of various address spaces. It is focused on memory management
+optimization and provides sophisticated accuracy/overhead handling mechanisms.
+Therefore, 'idle pages tracking' and 'perf mem' could provide a subset of
+DAMON's output, but cannot substitute DAMON.
+
+
+How can I optimize my system's memory management using DAMON?
+=============================================================
+
+Because there are several ways for the DAMON-based optimizations, we wrote a
+separate document, :doc:`/admin-guide/mm/damon/guide`. Please refer to that.
+
+
+Does DAMON support virtual memory only?
+=======================================
+
+No. The core of the DAMON is address space independent. The address space
+specific low level primitive parts including monitoring target regions
+constructions and actual access checks can be implemented and configured on the
+DAMON core by the users. In this way, DAMON users can monitor any address
+space with any access check technique.
+
+Nonetheless, DAMON provides vma tracking and PTE Accessed bit check based
+implementations of the address space dependent functions for the virtual memory
+by default, for a reference and convenient use. In near future, we will
+provide those for physical memory address space.
+
+
+Can I simply monitor page granularity?
+======================================
+
+Yes. You can do so by setting the ``min_nr_regions`` attribute higher than the
+working set size divided by the page size. Because the monitoring target
+regions size is forced to be ``>=page size``, the region split will make no
+effect.
diff --git a/Documentation/vm/damon/index.rst b/Documentation/vm/damon/index.rst
new file mode 100644
index 000000000000..17dca3c12aad
--- /dev/null
+++ b/Documentation/vm/damon/index.rst
@@ -0,0 +1,31 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========================
+DAMON: Data Access MONitor
+==========================
+
+DAMON is a data access monitoring framework subsystem for the Linux kernel.
+The core mechanisms of DAMON (refer to :doc:`design` for the detail) make it
+
+ - *accurate* (the monitoring output is useful enough for DRAM level memory
+ management; It might not appropriate for CPU Cache levels, though),
+ - *light-weight* (the monitoring overhead is low enough to be applied online),
+ and
+ - *scalable* (the upper-bound of the overhead is in constant range regardless
+ of the size of target workloads).
+
+Using this framework, therefore, the kernel's memory management mechanisms can
+make advanced decisions. Experimental memory management optimization works
+that incurring high data accesses monitoring overhead could implemented again.
+In user space, meanwhile, users who have some special workloads can write
+personalized applications for better understanding and optimizations of their
+workloads and systems.
+
+.. toctree::
+ :maxdepth: 2
+
+ faq
+ design
+ eval
+ api
+ plans
diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst
index eff5fbd492d0..b51f0d8992f8 100644
--- a/Documentation/vm/index.rst
+++ b/Documentation/vm/index.rst
@@ -32,6 +32,7 @@ descriptions of data structures and algorithms.
arch_pgtable_helpers
balance
cleancache
+ damon/index
free_page_reporting
frontswap
highmem
--
2.17.1
From: SeongJae Park <[email protected]>
This commit adds a simple user space tests for DAMON. The tests are
using kselftest framework.
Signed-off-by: SeongJae Park <[email protected]>
---
tools/testing/selftests/damon/Makefile | 7 +
.../selftests/damon/_chk_dependency.sh | 28 +++
tools/testing/selftests/damon/_chk_record.py | 109 ++++++++++++
.../testing/selftests/damon/debugfs_attrs.sh | 161 ++++++++++++++++++
.../testing/selftests/damon/debugfs_record.sh | 50 ++++++
5 files changed, 355 insertions(+)
create mode 100644 tools/testing/selftests/damon/Makefile
create mode 100644 tools/testing/selftests/damon/_chk_dependency.sh
create mode 100644 tools/testing/selftests/damon/_chk_record.py
create mode 100755 tools/testing/selftests/damon/debugfs_attrs.sh
create mode 100755 tools/testing/selftests/damon/debugfs_record.sh
diff --git a/tools/testing/selftests/damon/Makefile b/tools/testing/selftests/damon/Makefile
new file mode 100644
index 000000000000..cfd5393a4639
--- /dev/null
+++ b/tools/testing/selftests/damon/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+# Makefile for damon selftests
+
+TEST_FILES = _chk_dependency.sh _chk_record_file.py
+TEST_PROGS = debugfs_attrs.sh debugfs_record.sh
+
+include ../lib.mk
diff --git a/tools/testing/selftests/damon/_chk_dependency.sh b/tools/testing/selftests/damon/_chk_dependency.sh
new file mode 100644
index 000000000000..b304b7779976
--- /dev/null
+++ b/tools/testing/selftests/damon/_chk_dependency.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
+DBGFS=/sys/kernel/debug/damon
+
+if [ $EUID -ne 0 ];
+then
+ echo "Run as root"
+ exit $ksft_skip
+fi
+
+if [ ! -d $DBGFS ]
+then
+ echo "$DBGFS not found"
+ exit $ksft_skip
+fi
+
+for f in attrs record target_ids monitor_on
+do
+ if [ ! -f "$DBGFS/$f" ]
+ then
+ echo "$f not found"
+ exit 1
+ fi
+done
diff --git a/tools/testing/selftests/damon/_chk_record.py b/tools/testing/selftests/damon/_chk_record.py
new file mode 100644
index 000000000000..73e128904319
--- /dev/null
+++ b/tools/testing/selftests/damon/_chk_record.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"Check whether the DAMON record file is valid"
+
+import argparse
+import struct
+import sys
+
+fmt_version = 0
+
+def set_fmt_version(f):
+ global fmt_version
+
+ mark = f.read(16)
+ if mark == b'damon_recfmt_ver':
+ fmt_version = struct.unpack('i', f.read(4))[0]
+ else:
+ fmt_version = 0
+ f.seek(0)
+ return fmt_version
+
+def read_pid(f):
+ if fmt_version == 1:
+ pid = struct.unpack('i', f.read(4))[0]
+ else:
+ pid = struct.unpack('L', f.read(8))[0]
+
+def err_percent(val, expected):
+ return abs(val - expected) / expected * 100
+
+def chk_task_info(f):
+ pid = read_pid(f)
+ nr_regions = struct.unpack('I', f.read(4))[0]
+
+ if nr_regions > max_nr_regions:
+ print('too many regions: %d > %d' % (nr_regions, max_nr_regions))
+ exit(1)
+
+ nr_gaps = 0
+ eaddr = 0
+ for r in range(nr_regions):
+ saddr = struct.unpack('L', f.read(8))[0]
+ if eaddr and saddr != eaddr:
+ nr_gaps += 1
+ eaddr = struct.unpack('L', f.read(8))[0]
+ nr_accesses = struct.unpack('I', f.read(4))[0]
+
+ if saddr >= eaddr:
+ print('wrong region [%d,%d)' % (saddr, eaddr))
+ exit(1)
+
+ max_nr_accesses = aint / sint
+ if nr_accesses > max_nr_accesses:
+ if err_percent(nr_accesses, max_nr_accesses) > 15:
+ print('too high nr_access: expected %d but %d' %
+ (max_nr_accesses, nr_accesses))
+ exit(1)
+ if nr_gaps != 2:
+ print('number of gaps are not two but %d' % nr_gaps)
+ exit(1)
+
+def parse_time_us(bindat):
+ sec = struct.unpack('l', bindat[0:8])[0]
+ nsec = struct.unpack('l', bindat[8:16])[0]
+ return (sec * 1000000000 + nsec) / 1000
+
+def main():
+ global sint
+ global aint
+ global min_nr
+ global max_nr_regions
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument('file', metavar='<file>',
+ help='path to the record file')
+ parser.add_argument('--attrs', metavar='<attrs>',
+ default='5000 100000 1000000 10 1000',
+ help='content of debugfs attrs file')
+ args = parser.parse_args()
+ file_path = args.file
+ attrs = [int(x) for x in args.attrs.split()]
+ sint, aint, rint, min_nr, max_nr_regions = attrs
+
+ with open(file_path, 'rb') as f:
+ set_fmt_version(f)
+ last_aggr_time = None
+ while True:
+ timebin = f.read(16)
+ if len(timebin) != 16:
+ break
+
+ now = parse_time_us(timebin)
+ if not last_aggr_time:
+ last_aggr_time = now
+ else:
+ error = err_percent(now - last_aggr_time, aint)
+ if error > 15:
+ print('wrong aggr interval: expected %d, but %d' %
+ (aint, now - last_aggr_time))
+ exit(1)
+ last_aggr_time = now
+
+ nr_tasks = struct.unpack('I', f.read(4))[0]
+ for t in range(nr_tasks):
+ chk_task_info(f)
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/testing/selftests/damon/debugfs_attrs.sh b/tools/testing/selftests/damon/debugfs_attrs.sh
new file mode 100755
index 000000000000..c75557e8ba58
--- /dev/null
+++ b/tools/testing/selftests/damon/debugfs_attrs.sh
@@ -0,0 +1,161 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./_chk_dependency.sh
+
+# Test attrs file
+file="$DBGFS/attrs"
+
+ORIG_CONTENT=$(cat $file)
+
+echo 1 2 3 4 5 > $file
+if [ $? -ne 0 ]
+then
+ echo "$file write failed"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo 1 2 3 4 > $file
+if [ $? -eq 0 ]
+then
+ echo "$file write success (should failed)"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+CONTENT=$(cat $file)
+if [ "$CONTENT" != "1 2 3 4 5" ]
+then
+ echo "$file not written"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo $ORIG_CONTENT > $file
+
+# Test record file
+file="$DBGFS/record"
+
+ORIG_CONTENT=$(cat $file)
+
+echo "4242 foo.bar" > $file
+if [ $? -ne 0 ]
+then
+ echo "$file writing sane input failed"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo abc 2 3 > $file
+if [ $? -eq 0 ]
+then
+ echo "$file writing insane input 1 success (should failed)"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo 123 > $file
+if [ $? -eq 0 ]
+then
+ echo "$file writing insane input 2 success (should failed)"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+CONTENT=$(cat $file)
+if [ "$CONTENT" != "4242 foo.bar" ]
+then
+ echo "$file not written"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo "0 null" > $file
+if [ $? -ne 0 ]
+then
+ echo "$file disabling write fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+CONTENT=$(cat $file)
+if [ "$CONTENT" != "0 null" ]
+then
+ echo "$file not disabled"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo "4242 foo.bar" > $file
+if [ $? -ne 0 ]
+then
+ echo "$file writing sane data again fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo $ORIG_CONTENT > $file
+
+# Test target_ids file
+file="$DBGFS/target_ids"
+
+ORIG_CONTENT=$(cat $file)
+
+echo "1 2 3 4" > $file
+if [ $? -ne 0 ]
+then
+ echo "$file write fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo "1 2 abc 4" > $file
+if [ $? -ne 0 ]
+then
+ echo "$file write fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+CONTENT=$(cat $file)
+if [ "$CONTENT" != "1 2" ]
+then
+ echo "$file not written"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo abc 2 3 > $file
+if [ $? -ne 0 ]
+then
+ echo "$file wrong value write fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+if [ ! -z "$(cat $file)" ]
+then
+ echo "$file not cleared"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo > $file
+if [ $? -ne 0 ]
+then
+ echo "$file init fail"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+if [ ! -z "$(cat $file)" ]
+then
+ echo "$file not initialized"
+ echo $ORIG_CONTENT > $file
+ exit 1
+fi
+
+echo $ORIG_CONTENT > $file
+
+echo "PASS"
diff --git a/tools/testing/selftests/damon/debugfs_record.sh b/tools/testing/selftests/damon/debugfs_record.sh
new file mode 100755
index 000000000000..c0fb8d24dc32
--- /dev/null
+++ b/tools/testing/selftests/damon/debugfs_record.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./_chk_dependency.sh
+
+restore_attrs()
+{
+ echo $ORIG_ATTRS > $DBGFS/attrs
+ echo $ORIG_TARGET_IDS > $DBGFS/target_ids
+ echo $ORIG_RECORD > $DBGFS/record
+}
+
+ORIG_ATTRS=$(cat $DBGFS/attrs)
+ORIG_TARGET_IDS=$(cat $DBGFS/target_ids)
+ORIG_RECORD=$(cat $DBGFS/record)
+
+rfile=$pwd/damon.data
+
+rm -f $rfile
+ATTRS="5000 100000 1000000 10 1000"
+echo $ATTRS > $DBGFS/attrs
+echo 4096 $rfile > $DBGFS/record
+sleep 5 &
+echo $(pidof sleep) > $DBGFS/target_ids
+echo on > $DBGFS/monitor_on
+sleep 0.5
+killall sleep
+echo off > $DBGFS/monitor_on
+
+sync
+
+if [ ! -f $rfile ]
+then
+ echo "record file not made"
+ restore_attrs
+
+ exit 1
+fi
+
+python3 ./_chk_record.py $rfile --attrs "$ATTRS"
+if [ $? -ne 0 ]
+then
+ echo "record file is wrong"
+ restore_attrs
+ exit 1
+fi
+
+rm -f $rfile
+restore_attrs
+echo "PASS"
--
2.17.1
From: SeongJae Park <[email protected]>
'damon-dbgfs' provides simple user space interface for DAMON, but using
the interface for complex usages could require annoying repetitive
works. Writing a user space data access monitoring applications on top
of the debugfs interface and using the application would be better for
such complex cases.
This commit introduces a reference implementation of such user space
application built on top of the debugfs interface, namely 'DAMon
Operator' (DAMO). It contains a shallow wrapper python script of the
debugfs interface and various visualization of the monitoring results
convenient user interface.
Note that it is initially aimed to be used for minimal reference of the
'damon-dbgfs' interface and for debugging of the DAMON itself.
Signed-off-by: SeongJae Park <[email protected]>
---
tools/damon/.gitignore | 1 +
tools/damon/_damon.py | 130 ++++++++++++++
tools/damon/_dist.py | 35 ++++
tools/damon/_recfile.py | 23 +++
tools/damon/bin2txt.py | 67 +++++++
tools/damon/damo | 37 ++++
tools/damon/heats.py | 362 ++++++++++++++++++++++++++++++++++++++
tools/damon/nr_regions.py | 91 ++++++++++
tools/damon/record.py | 135 ++++++++++++++
tools/damon/report.py | 45 +++++
tools/damon/wss.py | 100 +++++++++++
11 files changed, 1026 insertions(+)
create mode 100644 tools/damon/.gitignore
create mode 100644 tools/damon/_damon.py
create mode 100644 tools/damon/_dist.py
create mode 100644 tools/damon/_recfile.py
create mode 100644 tools/damon/bin2txt.py
create mode 100755 tools/damon/damo
create mode 100644 tools/damon/heats.py
create mode 100644 tools/damon/nr_regions.py
create mode 100644 tools/damon/record.py
create mode 100644 tools/damon/report.py
create mode 100644 tools/damon/wss.py
diff --git a/tools/damon/.gitignore b/tools/damon/.gitignore
new file mode 100644
index 000000000000..96403d36ff93
--- /dev/null
+++ b/tools/damon/.gitignore
@@ -0,0 +1 @@
+__pycache__/*
diff --git a/tools/damon/_damon.py b/tools/damon/_damon.py
new file mode 100644
index 000000000000..1f6a292e8c25
--- /dev/null
+++ b/tools/damon/_damon.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Contains core functions for DAMON debugfs control.
+"""
+
+import os
+import subprocess
+
+debugfs_attrs = None
+debugfs_record = None
+debugfs_target_ids = None
+debugfs_monitor_on = None
+
+def set_target_id(tid):
+ with open(debugfs_target_ids, 'w') as f:
+ f.write('%s\n' % tid)
+
+def turn_damon(on_off):
+ return subprocess.call("echo %s > %s" % (on_off, debugfs_monitor_on),
+ shell=True, executable="/bin/bash")
+
+def is_damon_running():
+ with open(debugfs_monitor_on, 'r') as f:
+ return f.read().strip() == 'on'
+
+class Attrs:
+ sample_interval = None
+ aggr_interval = None
+ regions_update_interval = None
+ min_nr_regions = None
+ max_nr_regions = None
+ rbuf_len = None
+ rfile_path = None
+
+ def __init__(self, s, a, r, n, x, l, f):
+ self.sample_interval = s
+ self.aggr_interval = a
+ self.regions_update_interval = r
+ self.min_nr_regions = n
+ self.max_nr_regions = x
+ self.rbuf_len = l
+ self.rfile_path = f
+
+ def __str__(self):
+ return "%s %s %s %s %s %s %s" % (self.sample_interval,
+ self.aggr_interval, self.regions_update_interval,
+ self.min_nr_regions, self.max_nr_regions, self.rbuf_len,
+ self.rfile_path)
+
+ def attr_str(self):
+ return "%s %s %s %s %s " % (self.sample_interval, self.aggr_interval,
+ self.regions_update_interval, self.min_nr_regions,
+ self.max_nr_regions)
+
+ def record_str(self):
+ return '%s %s ' % (self.rbuf_len, self.rfile_path)
+
+ def apply(self):
+ ret = subprocess.call('echo %s > %s' % (self.attr_str(), debugfs_attrs),
+ shell=True, executable='/bin/bash')
+ if ret:
+ return ret
+ ret = subprocess.call('echo %s > %s' % (self.record_str(),
+ debugfs_record), shell=True, executable='/bin/bash')
+ if ret:
+ return ret
+
+def current_attrs():
+ with open(debugfs_attrs, 'r') as f:
+ attrs = f.read().split()
+ attrs = [int(x) for x in attrs]
+
+ with open(debugfs_record, 'r') as f:
+ rattrs = f.read().split()
+ attrs.append(int(rattrs[0]))
+ attrs.append(rattrs[1])
+
+ return Attrs(*attrs)
+
+def chk_update_debugfs(debugfs):
+ global debugfs_attrs
+ global debugfs_record
+ global debugfs_target_ids
+ global debugfs_monitor_on
+
+ debugfs_damon = os.path.join(debugfs, 'damon')
+ debugfs_attrs = os.path.join(debugfs_damon, 'attrs')
+ debugfs_record = os.path.join(debugfs_damon, 'record')
+ debugfs_target_ids = os.path.join(debugfs_damon, 'target_ids')
+ debugfs_monitor_on = os.path.join(debugfs_damon, 'monitor_on')
+
+ if not os.path.isdir(debugfs_damon):
+ print("damon debugfs dir (%s) not found", debugfs_damon)
+ exit(1)
+
+ for f in [debugfs_attrs, debugfs_record, debugfs_target_ids,
+ debugfs_monitor_on]:
+ if not os.path.isfile(f):
+ print("damon debugfs file (%s) not found" % f)
+ exit(1)
+
+def cmd_args_to_attrs(args):
+ "Generate attributes with specified arguments"
+ sample_interval = args.sample
+ aggr_interval = args.aggr
+ regions_update_interval = args.updr
+ min_nr_regions = args.minr
+ max_nr_regions = args.maxr
+ rbuf_len = args.rbuf
+ if not os.path.isabs(args.out):
+ args.out = os.path.join(os.getcwd(), args.out)
+ rfile_path = args.out
+ return Attrs(sample_interval, aggr_interval, regions_update_interval,
+ min_nr_regions, max_nr_regions, rbuf_len, rfile_path)
+
+def set_attrs_argparser(parser):
+ parser.add_argument('-d', '--debugfs', metavar='<debugfs>', type=str,
+ default='/sys/kernel/debug', help='debugfs mounted path')
+ parser.add_argument('-s', '--sample', metavar='<interval>', type=int,
+ default=5000, help='sampling interval')
+ parser.add_argument('-a', '--aggr', metavar='<interval>', type=int,
+ default=100000, help='aggregate interval')
+ parser.add_argument('-u', '--updr', metavar='<interval>', type=int,
+ default=1000000, help='regions update interval')
+ parser.add_argument('-n', '--minr', metavar='<# regions>', type=int,
+ default=10, help='minimal number of regions')
+ parser.add_argument('-m', '--maxr', metavar='<# regions>', type=int,
+ default=1000, help='maximum number of regions')
diff --git a/tools/damon/_dist.py b/tools/damon/_dist.py
new file mode 100644
index 000000000000..6435f98f4275
--- /dev/null
+++ b/tools/damon/_dist.py
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import os
+import struct
+import subprocess
+
+def access_patterns(f):
+ nr_regions = struct.unpack('I', f.read(4))[0]
+
+ patterns = []
+ for r in range(nr_regions):
+ saddr = struct.unpack('L', f.read(8))[0]
+ eaddr = struct.unpack('L', f.read(8))[0]
+ nr_accesses = struct.unpack('I', f.read(4))[0]
+ patterns.append([eaddr - saddr, nr_accesses])
+ return patterns
+
+def plot_dist(data_file, output_file, xlabel, ylabel):
+ terminal = output_file.split('.')[-1]
+ if not terminal in ['pdf', 'jpeg', 'png', 'svg']:
+ os.remove(data_file)
+ print("Unsupported plot output type.")
+ exit(-1)
+
+ gnuplot_cmd = """
+ set term %s;
+ set output '%s';
+ set key off;
+ set xlabel '%s';
+ set ylabel '%s';
+ plot '%s' with linespoints;""" % (terminal, output_file, xlabel, ylabel,
+ data_file)
+ subprocess.call(['gnuplot', '-e', gnuplot_cmd])
+ os.remove(data_file)
diff --git a/tools/damon/_recfile.py b/tools/damon/_recfile.py
new file mode 100644
index 000000000000..45dd8ffdb5ae
--- /dev/null
+++ b/tools/damon/_recfile.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import struct
+
+fmt_version = 0
+
+def set_fmt_version(f):
+ global fmt_version
+
+ mark = f.read(16)
+ if mark == b'damon_recfmt_ver':
+ fmt_version = struct.unpack('i', f.read(4))[0]
+ else:
+ fmt_version = 0
+ f.seek(0)
+ return fmt_version
+
+def target_id(f):
+ if fmt_version == 1:
+ return struct.unpack('i', f.read(4))[0]
+ else:
+ return struct.unpack('L', f.read(8))[0]
diff --git a/tools/damon/bin2txt.py b/tools/damon/bin2txt.py
new file mode 100644
index 000000000000..79516c72f449
--- /dev/null
+++ b/tools/damon/bin2txt.py
@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import argparse
+import os
+import struct
+import sys
+
+import _recfile
+
+def parse_time(bindat):
+ "bindat should be 16 bytes"
+ sec = struct.unpack('l', bindat[0:8])[0]
+ nsec = struct.unpack('l', bindat[8:16])[0]
+ return sec * 1000000000 + nsec;
+
+def pr_region(f):
+ saddr = struct.unpack('L', f.read(8))[0]
+ eaddr = struct.unpack('L', f.read(8))[0]
+ nr_accesses = struct.unpack('I', f.read(4))[0]
+ print("%012x-%012x(%10d):\t%d" %
+ (saddr, eaddr, eaddr - saddr, nr_accesses))
+
+def pr_task_info(f):
+ target_id = _recfile.target_id(f)
+ print("target_id: ", target_id)
+ nr_regions = struct.unpack('I', f.read(4))[0]
+ print("nr_regions: ", nr_regions)
+ for r in range(nr_regions):
+ pr_region(f)
+
+def set_argparser(parser):
+ parser.add_argument('--input', '-i', type=str, metavar='<file>',
+ default='damon.data', help='input file name')
+
+def main(args=None):
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ file_path = args.input
+
+ if not os.path.isfile(file_path):
+ print('input file (%s) is not exist' % file_path)
+ exit(1)
+
+ with open(file_path, 'rb') as f:
+ _recfile.set_fmt_version(f)
+ start_time = None
+ while True:
+ timebin = f.read(16)
+ if len(timebin) != 16:
+ break
+ time = parse_time(timebin)
+ if not start_time:
+ start_time = time
+ print("start_time: ", start_time)
+ print("rel time: %16d" % (time - start_time))
+ nr_tasks = struct.unpack('I', f.read(4))[0]
+ print("nr_tasks: ", nr_tasks)
+ for t in range(nr_tasks):
+ pr_task_info(f)
+ print("")
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/damon/damo b/tools/damon/damo
new file mode 100755
index 000000000000..58e1099ae5fc
--- /dev/null
+++ b/tools/damon/damo
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import argparse
+
+import record
+import report
+
+class SubCmdHelpFormatter(argparse.RawDescriptionHelpFormatter):
+ def _format_action(self, action):
+ parts = super(argparse.RawDescriptionHelpFormatter,
+ self)._format_action(action)
+ # skip sub parsers help
+ if action.nargs == argparse.PARSER:
+ parts = '\n'.join(parts.split('\n')[1:])
+ return parts
+
+parser = argparse.ArgumentParser(formatter_class=SubCmdHelpFormatter)
+
+subparser = parser.add_subparsers(title='command', dest='command',
+ metavar='<command>')
+subparser.required = True
+
+parser_record = subparser.add_parser('record',
+ help='record data accesses of the given target processes')
+record.set_argparser(parser_record)
+
+parser_report = subparser.add_parser('report',
+ help='report the recorded data accesses in the specified form')
+report.set_argparser(parser_report)
+
+args = parser.parse_args()
+
+if args.command == 'record':
+ record.main(args)
+elif args.command == 'report':
+ report.main(args)
diff --git a/tools/damon/heats.py b/tools/damon/heats.py
new file mode 100644
index 000000000000..78a2a793f50e
--- /dev/null
+++ b/tools/damon/heats.py
@@ -0,0 +1,362 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Transform binary trace data into human readable text that can be used for
+heatmap drawing, or directly plot the data in a heatmap format.
+
+Format of the text is:
+
+ <time> <space> <heat>
+ ...
+
+"""
+
+import argparse
+import os
+import struct
+import subprocess
+import sys
+import tempfile
+
+import _recfile
+
+class HeatSample:
+ space_idx = None
+ sz_time_space = None
+ heat = None
+
+ def __init__(self, space_idx, sz_time_space, heat):
+ if sz_time_space < 0:
+ raise RuntimeError()
+ self.space_idx = space_idx
+ self.sz_time_space = sz_time_space
+ self.heat = heat
+
+ def total_heat(self):
+ return self.heat * self.sz_time_space
+
+ def merge(self, sample):
+ "sample must have a space idx that same to self"
+ heat_sum = self.total_heat() + sample.total_heat()
+ self.heat = heat_sum / (self.sz_time_space + sample.sz_time_space)
+ self.sz_time_space += sample.sz_time_space
+
+def pr_samples(samples, time_idx, time_unit, region_unit):
+ display_time = time_idx * time_unit
+ for idx, sample in enumerate(samples):
+ display_addr = idx * region_unit
+ if not sample:
+ print("%s\t%s\t%s" % (display_time, display_addr, 0.0))
+ continue
+ print("%s\t%s\t%s" % (display_time, display_addr, sample.total_heat() /
+ time_unit / region_unit))
+
+def to_idx(value, min_, unit):
+ return (value - min_) // unit
+
+def read_task_heats(f, tid, aunit, amin, amax):
+ tid_ = _recfile.target_id(f)
+ nr_regions = struct.unpack('I', f.read(4))[0]
+ if tid_ != tid:
+ f.read(20 * nr_regions)
+ return None
+ samples = []
+ for i in range(nr_regions):
+ saddr = struct.unpack('L', f.read(8))[0]
+ eaddr = struct.unpack('L', f.read(8))[0]
+ eaddr = min(eaddr, amax - 1)
+ heat = struct.unpack('I', f.read(4))[0]
+
+ if eaddr <= amin:
+ continue
+ if saddr >= amax:
+ continue
+ saddr = max(amin, saddr)
+ eaddr = min(amax, eaddr)
+
+ sidx = to_idx(saddr, amin, aunit)
+ eidx = to_idx(eaddr - 1, amin, aunit)
+ for idx in range(sidx, eidx + 1):
+ sa = max(amin + idx * aunit, saddr)
+ ea = min(amin + (idx + 1) * aunit, eaddr)
+ sample = HeatSample(idx, (ea - sa), heat)
+ samples.append(sample)
+ return samples
+
+def parse_time(bindat):
+ sec = struct.unpack('l', bindat[0:8])[0]
+ nsec = struct.unpack('l', bindat[8:16])[0]
+ return sec * 1000000000 + nsec
+
+def apply_samples(target_samples, samples, start_time, end_time, aunit, amin):
+ for s in samples:
+ sample = HeatSample(s.space_idx,
+ s.sz_time_space * (end_time - start_time), s.heat)
+ idx = sample.space_idx
+ if not target_samples[idx]:
+ target_samples[idx] = sample
+ else:
+ target_samples[idx].merge(sample)
+
+def __pr_heats(f, tid, tunit, tmin, tmax, aunit, amin, amax):
+ heat_samples = [None] * ((amax - amin) // aunit)
+
+ start_time = 0
+ end_time = 0
+ last_flushed = -1
+ while True:
+ start_time = end_time
+ timebin = f.read(16)
+ if (len(timebin)) != 16:
+ break
+ end_time = parse_time(timebin)
+ nr_tasks = struct.unpack('I', f.read(4))[0]
+ samples_set = {}
+ for t in range(nr_tasks):
+ samples = read_task_heats(f, tid, aunit, amin, amax)
+ if samples:
+ samples_set[tid] = samples
+ if not tid in samples_set:
+ continue
+ if start_time >= tmax:
+ continue
+ if end_time <= tmin:
+ continue
+ start_time = max(start_time, tmin)
+ end_time = min(end_time, tmax)
+
+ sidx = to_idx(start_time, tmin, tunit)
+ eidx = to_idx(end_time - 1, tmin, tunit)
+ for idx in range(sidx, eidx + 1):
+ if idx != last_flushed:
+ pr_samples(heat_samples, idx, tunit, aunit)
+ heat_samples = [None] * ((amax - amin) // aunit)
+ last_flushed = idx
+ st = max(start_time, tmin + idx * tunit)
+ et = min(end_time, tmin + (idx + 1) * tunit)
+ apply_samples(heat_samples, samples_set[tid], st, et, aunit, amin)
+
+def pr_heats(args):
+ binfile = args.input
+ tid = args.tid
+ tres = args.tres
+ tmin = args.tmin
+ ares = args.ares
+ amin = args.amin
+
+ tunit = (args.tmax - tmin) // tres
+ aunit = (args.amax - amin) // ares
+
+ # Compensate the values so that those fit with the resolution
+ tmax = tmin + tunit * tres
+ amax = amin + aunit * ares
+
+ with open(binfile, 'rb') as f:
+ _recfile.set_fmt_version(f)
+ __pr_heats(f, tid, tunit, tmin, tmax, aunit, amin, amax)
+
+class GuideInfo:
+ tid = None
+ start_time = None
+ end_time = None
+ lowest_addr = None
+ highest_addr = None
+ gaps = None
+
+ def __init__(self, tid, start_time):
+ self.tid = tid
+ self.start_time = start_time
+ self.gaps = []
+
+ def regions(self):
+ regions = []
+ region = [self.lowest_addr]
+ for gap in self.gaps:
+ for idx, point in enumerate(gap):
+ if idx == 0:
+ region.append(point)
+ regions.append(region)
+ else:
+ region = [point]
+ region.append(self.highest_addr)
+ regions.append(region)
+ return regions
+
+ def total_space(self):
+ ret = 0
+ for r in self.regions():
+ ret += r[1] - r[0]
+ return ret
+
+ def __str__(self):
+ lines = ['target_id:%d' % self.tid]
+ lines.append('time: %d-%d (%d)' % (self.start_time, self.end_time,
+ self.end_time - self.start_time))
+ for idx, region in enumerate(self.regions()):
+ lines.append('region\t%2d: %020d-%020d (%d)' %
+ (idx, region[0], region[1], region[1] - region[0]))
+ return '\n'.join(lines)
+
+def is_overlap(region1, region2):
+ if region1[1] < region2[0]:
+ return False
+ if region2[1] < region1[0]:
+ return False
+ return True
+
+def overlap_region_of(region1, region2):
+ return [max(region1[0], region2[0]), min(region1[1], region2[1])]
+
+def overlapping_regions(regions1, regions2):
+ overlap_regions = []
+ for r1 in regions1:
+ for r2 in regions2:
+ if is_overlap(r1, r2):
+ r1 = overlap_region_of(r1, r2)
+ if r1:
+ overlap_regions.append(r1)
+ return overlap_regions
+
+def get_guide_info(binfile):
+ "Read file, return the set of guide information objects of the data"
+ guides = {}
+ with open(binfile, 'rb') as f:
+ _recfile.set_fmt_version(f)
+ while True:
+ timebin = f.read(16)
+ if len(timebin) != 16:
+ break
+ monitor_time = parse_time(timebin)
+ nr_tasks = struct.unpack('I', f.read(4))[0]
+ for t in range(nr_tasks):
+ tid = _recfile.target_id(f)
+ nr_regions = struct.unpack('I', f.read(4))[0]
+ if not tid in guides:
+ guides[tid] = GuideInfo(tid, monitor_time)
+ guide = guides[tid]
+ guide.end_time = monitor_time
+
+ last_addr = None
+ gaps = []
+ for r in range(nr_regions):
+ saddr = struct.unpack('L', f.read(8))[0]
+ eaddr = struct.unpack('L', f.read(8))[0]
+ f.read(4)
+
+ if not guide.lowest_addr or saddr < guide.lowest_addr:
+ guide.lowest_addr = saddr
+ if not guide.highest_addr or eaddr > guide.highest_addr:
+ guide.highest_addr = eaddr
+
+ if not last_addr:
+ last_addr = eaddr
+ continue
+ if last_addr != saddr:
+ gaps.append([last_addr, saddr])
+ last_addr = eaddr
+
+ if not guide.gaps:
+ guide.gaps = gaps
+ else:
+ guide.gaps = overlapping_regions(guide.gaps, gaps)
+ return sorted(list(guides.values()), key=lambda x: x.total_space(),
+ reverse=True)
+
+def pr_guide(binfile):
+ for guide in get_guide_info(binfile):
+ print(guide)
+
+def region_sort_key(region):
+ return region[1] - region[0]
+
+def set_missed_args(args):
+ if args.tid and args.tmin and args.tmax and args.amin and args.amax:
+ return
+ guides = get_guide_info(args.input)
+ guide = guides[0]
+ if not args.tid:
+ args.tid = guide.tid
+ for g in guides:
+ if g.tid == args.tid:
+ guide = g
+ break
+
+ if not args.tmin:
+ args.tmin = guide.start_time
+ if not args.tmax:
+ args.tmax = guide.end_time
+
+ if not args.amin or not args.amax:
+ region = sorted(guide.regions(), key=lambda x: x[1] - x[0],
+ reverse=True)[0]
+ args.amin = region[0]
+ args.amax = region[1]
+
+def plot_heatmap(data_file, output_file):
+ terminal = output_file.split('.')[-1]
+ if not terminal in ['pdf', 'jpeg', 'png', 'svg']:
+ os.remove(data_file)
+ print("Unsupported plot output type.")
+ exit(-1)
+
+ gnuplot_cmd = """
+ set term %s;
+ set output '%s';
+ set key off;
+ set xrange [0:];
+ set yrange [0:];
+ set xlabel 'Time (ns)';
+ set ylabel 'Address (bytes)';
+ plot '%s' using 1:2:3 with image;""" % (terminal, output_file, data_file)
+ subprocess.call(['gnuplot', '-e', gnuplot_cmd])
+ os.remove(data_file)
+
+def set_argparser(parser):
+ parser.add_argument('--input', '-i', type=str, metavar='<file>',
+ default='damon.data', help='input file name')
+ parser.add_argument('--tid', metavar='<id>', type=int,
+ help='target id')
+ parser.add_argument('--tres', metavar='<resolution>', type=int,
+ default=500, help='time resolution of the output')
+ parser.add_argument('--tmin', metavar='<time>', type=lambda x: int(x,0),
+ help='minimal time of the output')
+ parser.add_argument('--tmax', metavar='<time>', type=lambda x: int(x,0),
+ help='maximum time of the output')
+ parser.add_argument('--ares', metavar='<resolution>', type=int, default=500,
+ help='space address resolution of the output')
+ parser.add_argument('--amin', metavar='<address>', type=lambda x: int(x,0),
+ help='minimal space address of the output')
+ parser.add_argument('--amax', metavar='<address>', type=lambda x: int(x,0),
+ help='maximum space address of the output')
+ parser.add_argument('--guide', action='store_true',
+ help='print a guidance for the min/max/resolution settings')
+ parser.add_argument('--heatmap', metavar='<file>', type=str,
+ help='heatmap image file to create')
+
+def main(args=None):
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ if args.guide:
+ pr_guide(args.input)
+ else:
+ set_missed_args(args)
+ orig_stdout = sys.stdout
+ if args.heatmap:
+ tmp_path = tempfile.mkstemp()[1]
+ tmp_file = open(tmp_path, 'w')
+ sys.stdout = tmp_file
+
+ pr_heats(args)
+
+ if args.heatmap:
+ sys.stdout = orig_stdout
+ tmp_file.flush()
+ tmp_file.close()
+ plot_heatmap(tmp_path, args.heatmap)
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/damon/nr_regions.py b/tools/damon/nr_regions.py
new file mode 100644
index 000000000000..960dd4362472
--- /dev/null
+++ b/tools/damon/nr_regions.py
@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"Print out distribution of the number of regions in the given record"
+
+import argparse
+import struct
+import sys
+import tempfile
+
+import _dist
+import _recfile
+
+def set_argparser(parser):
+ parser.add_argument('--input', '-i', type=str, metavar='<file>',
+ default='damon.data', help='input file name')
+ parser.add_argument('--range', '-r', type=int, nargs=3,
+ metavar=('<start>', '<stop>', '<step>'),
+ help='range of percentiles to print')
+ parser.add_argument('--sortby', '-s', choices=['time', 'size'],
+ help='the metric to be used for sorting the number of regions')
+ parser.add_argument('--plot', '-p', type=str, metavar='<file>',
+ help='plot the distribution to an image file')
+
+def main(args=None):
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ percentiles = [0, 25, 50, 75, 100]
+
+ file_path = args.input
+ if args.range:
+ percentiles = range(args.range[0], args.range[1], args.range[2])
+ nr_regions_sort = True
+ if args.sortby == 'time':
+ nr_regions_sort = False
+
+ tid_pattern_map = {}
+ with open(file_path, 'rb') as f:
+ _recfile.set_fmt_version(f)
+ start_time = None
+ while True:
+ timebin = f.read(16)
+ if len(timebin) != 16:
+ break
+ nr_tasks = struct.unpack('I', f.read(4))[0]
+ for t in range(nr_tasks):
+ tid = _recfile.target_id(f)
+ if not tid in tid_pattern_map:
+ tid_pattern_map[tid] = []
+ tid_pattern_map[tid].append(_dist.access_patterns(f))
+
+ orig_stdout = sys.stdout
+ if args.plot:
+ tmp_path = tempfile.mkstemp()[1]
+ tmp_file = open(tmp_path, 'w')
+ sys.stdout = tmp_file
+
+ print('# <percentile> <# regions>')
+ for tid in tid_pattern_map.keys():
+ # Skip firs 20 regions as those would not adaptively adjusted
+ snapshots = tid_pattern_map[tid][20:]
+ nr_regions_dist = []
+ for snapshot in snapshots:
+ nr_regions_dist.append(len(snapshot))
+ if nr_regions_sort:
+ nr_regions_dist.sort(reverse=False)
+
+ print('# target_id\t%s' % tid)
+ print('# avr:\t%d' % (sum(nr_regions_dist) / len(nr_regions_dist)))
+ for percentile in percentiles:
+ thres_idx = int(percentile / 100.0 * len(nr_regions_dist))
+ if thres_idx == len(nr_regions_dist):
+ thres_idx -= 1
+ threshold = nr_regions_dist[thres_idx]
+ print('%d\t%d' % (percentile, nr_regions_dist[thres_idx]))
+
+ if args.plot:
+ sys.stdout = orig_stdout
+ tmp_file.flush()
+ tmp_file.close()
+ xlabel = 'runtime (percent)'
+ if nr_regions_sort:
+ xlabel = 'percentile'
+ _dist.plot_dist(tmp_path, args.plot, xlabel,
+ 'number of monitoring target regions')
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/damon/record.py b/tools/damon/record.py
new file mode 100644
index 000000000000..6d1cbe593b94
--- /dev/null
+++ b/tools/damon/record.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Record data access patterns of the target process.
+"""
+
+import argparse
+import os
+import signal
+import subprocess
+import time
+
+import _damon
+
+def pidfd_open(pid):
+ import ctypes
+ libc = ctypes.CDLL(None)
+ syscall = libc.syscall
+ syscall.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.c_uint]
+ syscall.restype = ctypes.c_long
+
+ NR_pidfd_open = 434
+
+ return syscall(NR_pidfd_open, pid, 0)
+
+def do_record(target, is_target_cmd, attrs, old_attrs, pidfd):
+ if os.path.isfile(attrs.rfile_path):
+ os.rename(attrs.rfile_path, attrs.rfile_path + '.old')
+
+ if attrs.apply():
+ print('attributes (%s) failed to be applied' % attrs)
+ cleanup_exit(old_attrs, -1)
+ print('# damon attrs: %s %s' % (attrs.attr_str(), attrs.record_str()))
+ if is_target_cmd:
+ p = subprocess.Popen(target, shell=True, executable='/bin/bash')
+ target = p.pid
+
+ if pidfd:
+ fd = pidfd_open(int(target))
+ if fd < 0:
+ print('failed getting pidfd of %s: %s' % (target, fd))
+ cleanup_exit(old_attrs, -1)
+
+ # NOTE: The race is still possible because the pid might be already
+ # reused before above pidfd_open() returned. Eliminating the race is
+ # impossible unless we drop the pid support. This option handling is
+ # only for reference of the pidfd usage.
+ target = 'pidfd %s' % fd
+
+ if _damon.set_target_id(target):
+ print('target id setting (%s) failed' % target)
+ cleanup_exit(old_attrs, -2)
+ if _damon.turn_damon('on'):
+ print('could not turn on damon' % target)
+ cleanup_exit(old_attrs, -3)
+ while not _damon.is_damon_running():
+ time.sleep(1)
+ print('Press Ctrl+C to stop')
+ if is_target_cmd:
+ p.wait()
+ while True:
+ # damon will turn it off by itself if the target tasks are terminated.
+ if not _damon.is_damon_running():
+ break
+ time.sleep(1)
+
+ if pidfd:
+ os.close(fd)
+ cleanup_exit(old_attrs, 0)
+
+def cleanup_exit(orig_attrs, exit_code):
+ if _damon.is_damon_running():
+ if _damon.turn_damon('off'):
+ print('failed to turn damon off!')
+ while _damon.is_damon_running():
+ time.sleep(1)
+ if orig_attrs:
+ if orig_attrs.apply():
+ print('original attributes (%s) restoration failed!' % orig_attrs)
+ exit(exit_code)
+
+def sighandler(signum, frame):
+ print('\nsignal %s received' % signum)
+ cleanup_exit(orig_attrs, signum)
+
+def chk_permission():
+ if os.geteuid() != 0:
+ print("Run as root")
+ exit(1)
+
+def set_argparser(parser):
+ _damon.set_attrs_argparser(parser)
+ parser.add_argument('target', type=str, metavar='<target>',
+ help='the target command or the pid to record')
+ parser.add_argument('--pidfd', action='store_true',
+ help='use pidfd type target id')
+ parser.add_argument('-l', '--rbuf', metavar='<len>', type=int,
+ default=1024*1024, help='length of record result buffer')
+ parser.add_argument('-o', '--out', metavar='<file path>', type=str,
+ default='damon.data', help='output file path')
+
+def main(args=None):
+ global orig_attrs
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ chk_permission()
+ _damon.chk_update_debugfs(args.debugfs)
+
+ signal.signal(signal.SIGINT, sighandler)
+ signal.signal(signal.SIGTERM, sighandler)
+ orig_attrs = _damon.current_attrs()
+
+ args.schemes = ''
+ pidfd = args.pidfd
+ new_attrs = _damon.cmd_args_to_attrs(args)
+ target = args.target
+
+ target_fields = target.split()
+ if not subprocess.call('which %s &> /dev/null' % target_fields[0],
+ shell=True, executable='/bin/bash'):
+ do_record(target, True, new_attrs, orig_attrs, pidfd)
+ else:
+ try:
+ pid = int(target)
+ except:
+ print('target \'%s\' is neither a command, nor a pid' % target)
+ exit(1)
+ do_record(target, False, new_attrs, orig_attrs, pidfd)
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/damon/report.py b/tools/damon/report.py
new file mode 100644
index 000000000000..c661c7b2f1af
--- /dev/null
+++ b/tools/damon/report.py
@@ -0,0 +1,45 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import argparse
+
+import bin2txt
+import heats
+import nr_regions
+import wss
+
+def set_argparser(parser):
+ subparsers = parser.add_subparsers(title='report type', dest='report_type',
+ metavar='<report type>', help='the type of the report to generate')
+ subparsers.required = True
+
+ parser_raw = subparsers.add_parser('raw', help='human readable raw data')
+ bin2txt.set_argparser(parser_raw)
+
+ parser_heats = subparsers.add_parser('heats', help='heats of regions')
+ heats.set_argparser(parser_heats)
+
+ parser_wss = subparsers.add_parser('wss', help='working set size')
+ wss.set_argparser(parser_wss)
+
+ parser_nr_regions = subparsers.add_parser('nr_regions',
+ help='number of regions')
+ nr_regions.set_argparser(parser_nr_regions)
+
+def main(args=None):
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ if args.report_type == 'raw':
+ bin2txt.main(args)
+ elif args.report_type == 'heats':
+ heats.main(args)
+ elif args.report_type == 'wss':
+ wss.main(args)
+ elif args.report_type == 'nr_regions':
+ nr_regions.main(args)
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/damon/wss.py b/tools/damon/wss.py
new file mode 100644
index 000000000000..54e2ac7cf83b
--- /dev/null
+++ b/tools/damon/wss.py
@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"Print out the distribution of the working set sizes of the given trace"
+
+import argparse
+import struct
+import sys
+import tempfile
+
+import _dist
+import _recfile
+
+def set_argparser(parser):
+ parser.add_argument('--input', '-i', type=str, metavar='<file>',
+ default='damon.data', help='input file name')
+ parser.add_argument('--range', '-r', type=int, nargs=3,
+ metavar=('<start>', '<stop>', '<step>'),
+ help='range of wss percentiles to print')
+ parser.add_argument('--thres', '-t', type=int, default=1,
+ metavar='<# accesses>',
+ help='minimal number of accesses for treated as working set')
+ parser.add_argument('--sortby', '-s', choices=['time', 'size'],
+ help='the metric to be used for the sort of the working set sizes')
+ parser.add_argument('--plot', '-p', type=str, metavar='<file>',
+ help='plot the distribution to an image file')
+
+def main(args=None):
+ if not args:
+ parser = argparse.ArgumentParser()
+ set_argparser(parser)
+ args = parser.parse_args()
+
+ percentiles = [0, 25, 50, 75, 100]
+
+ file_path = args.input
+ if args.range:
+ percentiles = range(args.range[0], args.range[1], args.range[2])
+ wss_sort = True
+ if args.sortby == 'time':
+ wss_sort = False
+
+ tid_pattern_map = {}
+ with open(file_path, 'rb') as f:
+ _recfile.set_fmt_version(f)
+ start_time = None
+ while True:
+ timebin = f.read(16)
+ if len(timebin) != 16:
+ break
+ nr_tasks = struct.unpack('I', f.read(4))[0]
+ for t in range(nr_tasks):
+ tid = _recfile.target_id(f)
+ if not tid in tid_pattern_map:
+ tid_pattern_map[tid] = []
+ tid_pattern_map[tid].append(_dist.access_patterns(f))
+
+ orig_stdout = sys.stdout
+ if args.plot:
+ tmp_path = tempfile.mkstemp()[1]
+ tmp_file = open(tmp_path, 'w')
+ sys.stdout = tmp_file
+
+ print('# <percentile> <wss>')
+ for tid in tid_pattern_map.keys():
+ # Skip first 20 snapshots as regions may not adjusted yet.
+ snapshots = tid_pattern_map[tid][20:]
+ wss_dist = []
+ for snapshot in snapshots:
+ wss = 0
+ for p in snapshot:
+ # Ignore regions not accessed
+ if p[1] < args.thres:
+ continue
+ wss += p[0]
+ wss_dist.append(wss)
+ if wss_sort:
+ wss_dist.sort(reverse=False)
+
+ print('# target_id\t%s' % tid)
+ print('# avr:\t%d' % (sum(wss_dist) / len(wss_dist)))
+ for percentile in percentiles:
+ thres_idx = int(percentile / 100.0 * len(wss_dist))
+ if thres_idx == len(wss_dist):
+ thres_idx -= 1
+ threshold = wss_dist[thres_idx]
+ print('%d\t%d' % (percentile, wss_dist[thres_idx]))
+
+ if args.plot:
+ sys.stdout = orig_stdout
+ tmp_file.flush()
+ tmp_file.close()
+ xlabel = 'runtime (percent)'
+ if wss_sort:
+ xlabel = 'percentile'
+ _dist.plot_dist(tmp_path, args.plot, xlabel,
+ 'working set size (bytes)')
+
+if __name__ == '__main__':
+ main()
--
2.17.1
From: SeongJae Park <[email protected]>
DAMON is a data access monitoring framework for the Linux kernel. The
core mechanisms of DAMON make it
- accurate (the monitoring output is useful enough for DRAM level
performance-centric memory management; It might be inappropriate for
CPU cache levels, though),
- light-weight (the monitoring overhead is normally low enough to be
applied online), and
- scalable (the upper-bound of the overhead is in constant range
regardless of the size of target workloads).
Using this framework, hence, we can easily write efficient kernel space
data access monitoring applications. For example, the kernel's memory
management mechanisms can make advanced decisions using this.
Experimental data access aware optimization works that incurring high
access monitoring overhead could implemented again on top of this.
Due to its simple and flexible interface, providing user space interface
would be also easy. Then, user space users who have some special
workloads can write personalized applications for better understanding
and optimizations of their workloads and systems.
---
Nevertheless, this commit is defining and implementing only basic access
check part without the overhead-accuracy handling core logic. The basic
access check is as below.
The output of DAMON says what memory regions are how frequently accessed
for a given duration. The resolution of the access frequency is
controlled by setting ``sampling interval`` and ``aggregation
interval``. In detail, DAMON checks access to each page per ``sampling
interval`` and aggregates the results. In other words, counts the
number of the accesses to each region. After each ``aggregation
interval`` passes, DAMON calls callback functions that previously
registered by users so that users can read the aggregated results and
then clears the results. This can be described in below simple
pseudo-code::
while monitoring_on:
for page in monitoring_target:
if accessed(page):
nr_accesses[page] += 1
if time() % aggregation_interval == 0:
for callback in user_registered_callbacks:
callback(monitoring_target, nr_accesses)
for page in monitoring_target:
nr_accesses[page] = 0
sleep(sampling interval)
The target regions constructed at the beginning of the monitoring and
updated after each ``regions_update_interval``, because the target
regions could be dynamically changed (e.g., mmap() or memory hotplug).
The monitoring overhead of this mechanism will arbitrarily increase as
the size of the target workload grows.
The basic monitoring primitives for actual access check and dynamic
target regions construction aren't in the core part of DAMON. Instead,
it allows users to implement their own primitives that optimized for
their use case and configure DAMON to use those. In other words, users
cannot use current version of DAMON without some additional works.
Following commits will implement the core mechanisms for the
overhead-accuracy control and default primitives implementations.
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Leonard Foerster <[email protected]>
---
include/linux/damon.h | 167 ++++++++++++++++++++++
mm/Kconfig | 2 +
mm/Makefile | 1 +
mm/damon/Kconfig | 15 ++
mm/damon/Makefile | 3 +
mm/damon/core.c | 316 ++++++++++++++++++++++++++++++++++++++++++
6 files changed, 504 insertions(+)
create mode 100644 include/linux/damon.h
create mode 100644 mm/damon/Kconfig
create mode 100644 mm/damon/Makefile
create mode 100644 mm/damon/core.c
diff --git a/include/linux/damon.h b/include/linux/damon.h
new file mode 100644
index 000000000000..387fa4399fc8
--- /dev/null
+++ b/include/linux/damon.h
@@ -0,0 +1,167 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * DAMON api
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#ifndef _DAMON_H_
+#define _DAMON_H_
+
+#include <linux/mutex.h>
+#include <linux/time64.h>
+#include <linux/types.h>
+
+struct damon_ctx;
+
+/**
+ * struct damon_primitive Monitoring primitives for given use cases.
+ *
+ * @init_target_regions: Constructs initial monitoring target regions.
+ * @update_target_regions: Updates monitoring target regions.
+ * @prepare_access_checks: Prepares next access check of target regions.
+ * @check_accesses: Checks the access of target regions.
+ * @reset_aggregated: Resets aggregated accesses monitoring results.
+ * @target_valid: Determine if the target is valid.
+ * @cleanup: Cleans up the context.
+ *
+ * DAMON can be extended for various address spaces and usages. For this,
+ * users should register the low level primitives for their target address
+ * space and usecase via the &damon_ctx.primitive. Then, the monitoring thread
+ * calls @init_target_regions and @prepare_access_checks before starting the
+ * monitoring, @update_target_regions after each
+ * &damon_ctx.regions_update_interval, and @check_accesses, @target_valid and
+ * @prepare_access_checks after each &damon_ctx.sample_interval. Finally,
+ * @reset_aggregated is called after each &damon_ctx.aggr_interval.
+ *
+ * @init_target_regions should construct proper monitoring target regions and
+ * link those to the DAMON context struct. The regions should be defined by
+ * user and saved in @damon_ctx.target.
+ * @update_target_regions should update the monitoring target regions for
+ * current status.
+ * @prepare_access_checks should manipulate the monitoring regions to be
+ * prepared for the next access check.
+ * @check_accesses should check the accesses to each region that made after the
+ * last preparation and update the number of observed accesses of each region.
+ * @reset_aggregated should reset the access monitoring results that aggregated
+ * by @check_accesses.
+ * @target_valid should check whether the target is still valid for the
+ * monitoring.
+ * @cleanup is called from @kdamond just before its termination. After this
+ * call, only @kdamond_lock and @kdamond will be touched.
+ */
+struct damon_primitive {
+ void (*init_target_regions)(struct damon_ctx *context);
+ void (*update_target_regions)(struct damon_ctx *context);
+ void (*prepare_access_checks)(struct damon_ctx *context);
+ void (*check_accesses)(struct damon_ctx *context);
+ void (*reset_aggregated)(struct damon_ctx *context);
+ bool (*target_valid)(void *target);
+ void (*cleanup)(struct damon_ctx *context);
+};
+
+/*
+ * struct damon_callback Monitoring events notification callbacks.
+ *
+ * @before_start: Called before starting the monitoring.
+ * @after_sampling: Called after each sampling.
+ * @after_aggregation: Called after each aggregation.
+ * @before_terminate: Called before terminating the monitoring.
+ * @private: User private data.
+ *
+ * The monitoring thread (&damon_ctx->kdamond) calls @before_start and
+ * @before_terminate just before starting and finishing the monitoring,
+ * respectively. Therefore, those are good places for installing and cleaning
+ * @private.
+ *
+ * The monitoring thread calls @after_sampling and @after_aggregation for each
+ * of the sampling intervals and aggregation intervals, respectively.
+ * Therefore, users can safely access the monitoring results without additional
+ * protection. For the reason, users are recommended to use these callback for
+ * the accesses to the results.
+ *
+ * If any callback returns non-zero, monitoring stops.
+ */
+struct damon_callback {
+ void *private;
+
+ int (*before_start)(struct damon_ctx *context);
+ int (*after_sampling)(struct damon_ctx *context);
+ int (*after_aggregation)(struct damon_ctx *context);
+ int (*before_terminate)(struct damon_ctx *context);
+};
+
+/**
+ * struct damon_ctx - Represents a context for each monitoring. This is the
+ * main interface that allows users to set the attributes and get the results
+ * of the monitoring.
+ *
+ * @sample_interval: The time between access samplings.
+ * @aggr_interval: The time between monitor results aggregations.
+ * @regions_update_interval: The time between monitor regions updates.
+ *
+ * For each @sample_interval, DAMON checks whether each region is accessed or
+ * not. It aggregates and keeps the access information (number of accesses to
+ * each region) for @aggr_interval time. DAMON also checks whether the target
+ * memory regions need update (e.g., by ``mmap()`` calls from the application,
+ * in case of virtual memory monitoring) and applies the changes for each
+ * @regions_update_interval. All time intervals are in micro-seconds. Please
+ * refer to &struct damon_primitive and &struct damon_callback for more detail.
+ *
+ * @kdamond: Kernel thread who does the monitoring.
+ * @kdamond_stop: Notifies whether kdamond should stop.
+ * @kdamond_lock: Mutex for the synchronizations with @kdamond.
+ *
+ * For each monitoring context, one kernel thread for the monitoring is
+ * created. The pointer to the thread is stored in @kdamond.
+ *
+ * Once started, the monitoring thread runs until explicitly required to be
+ * terminated or every monitoring target is invalid. The validity of the
+ * targets is checked via the &damon_primitive.target_valid of @primitive. The
+ * termination can also be explicitly requested by writing non-zero to
+ * @kdamond_stop. The thread sets @kdamond to NULL when it terminates.
+ * Therefore, users can know whether the monitoring is ongoing or terminated by
+ * reading @kdamond. Reads and writes to @kdamond and @kdamond_stop from
+ * outside of the monitoring thread must be protected by @kdamond_lock.
+ *
+ * Note that the monitoring thread protects only @kdamond and @kdamond_stop via
+ * @kdamond_lock. Accesses to other fields must be protected by themselves.
+ *
+ * @primitive: Set of monitoring primitives for given use cases.
+ * @callback: Set of callbacks for monitoring events notifications.
+ *
+ * @target: Pointer to the user-defined monitoring target.
+ */
+struct damon_ctx {
+ unsigned long sample_interval;
+ unsigned long aggr_interval;
+ unsigned long regions_update_interval;
+
+/* private */
+ struct timespec64 last_aggregation;
+ struct timespec64 last_regions_update;
+
+/* public */
+ struct task_struct *kdamond;
+ bool kdamond_stop;
+ struct mutex kdamond_lock;
+
+ struct damon_primitive primitive;
+ struct damon_callback callback;
+
+ void *target;
+};
+
+#ifdef CONFIG_DAMON
+
+struct damon_ctx *damon_new_ctx(void);
+void damon_destroy_ctx(struct damon_ctx *ctx);
+int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
+ unsigned long aggr_int, unsigned long regions_update_int);
+
+int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
+int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
+
+#endif /* CONFIG_DAMON */
+
+#endif /* _DAMON_H */
diff --git a/mm/Kconfig b/mm/Kconfig
index 390165ffbb0f..b97f2e8ab83f 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -859,4 +859,6 @@ config ARCH_HAS_HUGEPD
config MAPPING_DIRTY_HELPERS
bool
+source "mm/damon/Kconfig"
+
endmenu
diff --git a/mm/Makefile b/mm/Makefile
index d73aed0fc99c..8022b8f04096 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -120,3 +120,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o
obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
+obj-$(CONFIG_DAMON) += damon/
diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
new file mode 100644
index 000000000000..d00e99ac1a15
--- /dev/null
+++ b/mm/damon/Kconfig
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+menu "Data Access Monitoring"
+
+config DAMON
+ bool "DAMON: Data Access Monitoring Framework"
+ help
+ This builds a framework that allows kernel subsystems to monitor
+ access frequency of each memory region. The information can be useful
+ for performance-centric DRAM level memory management.
+
+ See https://damonitor.github.io/doc/html/latest-damon/index.html for
+ more information.
+
+endmenu
diff --git a/mm/damon/Makefile b/mm/damon/Makefile
new file mode 100644
index 000000000000..4fd2edb4becf
--- /dev/null
+++ b/mm/damon/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_DAMON) := core.o
diff --git a/mm/damon/core.c b/mm/damon/core.c
new file mode 100644
index 000000000000..8963804efdf9
--- /dev/null
+++ b/mm/damon/core.c
@@ -0,0 +1,316 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Data Access Monitor
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#define pr_fmt(fmt) "damon: " fmt
+
+#include <linux/damon.h>
+#include <linux/delay.h>
+#include <linux/kthread.h>
+#include <linux/slab.h>
+
+static DEFINE_MUTEX(damon_lock);
+static int nr_running_ctxs;
+
+struct damon_ctx *damon_new_ctx(void)
+{
+ struct damon_ctx *ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ ctx->sample_interval = 5 * 1000;
+ ctx->aggr_interval = 100 * 1000;
+ ctx->regions_update_interval = 1000 * 1000;
+
+ ktime_get_coarse_ts64(&ctx->last_aggregation);
+ ctx->last_regions_update = ctx->last_aggregation;
+
+ mutex_init(&ctx->kdamond_lock);
+
+ ctx->target = NULL;
+
+ return ctx;
+}
+
+void damon_destroy_ctx(struct damon_ctx *ctx)
+{
+ kfree(ctx);
+}
+
+/**
+ * damon_set_attrs() - Set attributes for the monitoring.
+ * @ctx: monitoring context
+ * @sample_int: time interval between samplings
+ * @aggr_int: time interval between aggregations
+ * @regions_update_int: time interval between target regions update
+ *
+ * This function should not be called while the kdamond is running.
+ * Every time interval is in micro-seconds.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
+ unsigned long aggr_int, unsigned long regions_update_int)
+{
+ ctx->sample_interval = sample_int;
+ ctx->aggr_interval = aggr_int;
+ ctx->regions_update_interval = regions_update_int;
+
+ return 0;
+}
+
+static bool damon_kdamond_running(struct damon_ctx *ctx)
+{
+ bool running;
+
+ mutex_lock(&ctx->kdamond_lock);
+ running = ctx->kdamond != NULL;
+ mutex_unlock(&ctx->kdamond_lock);
+
+ return running;
+}
+
+static int kdamond_fn(void *data);
+
+/*
+ * __damon_start() - Starts monitoring with given context.
+ * @ctx: monitoring context
+ *
+ * This function should be called while damon_lock is hold.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+static int __damon_start(struct damon_ctx *ctx)
+{
+ int err = -EBUSY;
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (!ctx->kdamond) {
+ err = 0;
+ ctx->kdamond_stop = false;
+ ctx->kdamond = kthread_create(kdamond_fn, ctx, "kdamond.%d",
+ nr_running_ctxs);
+ if (IS_ERR(ctx->kdamond))
+ err = PTR_ERR(ctx->kdamond);
+ else
+ wake_up_process(ctx->kdamond);
+ }
+ mutex_unlock(&ctx->kdamond_lock);
+
+ return err;
+}
+
+/**
+ * damon_start() - Starts the monitorings for a given group of contexts.
+ * @ctxs: an array of the pointers for contexts to start monitoring
+ * @nr_ctxs: size of @ctxs
+ *
+ * This function starts a group of monitoring threads for a group of monitoring
+ * contexts. One thread per each context is created and run in parallel. The
+ * caller should handle synchronization between the threads by itself. If a
+ * group of threads that created by other 'damon_start()' call is currently
+ * running, this function does nothing but returns -EBUSY.
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int damon_start(struct damon_ctx **ctxs, int nr_ctxs)
+{
+ int i;
+ int err = 0;
+
+ mutex_lock(&damon_lock);
+ if (nr_running_ctxs) {
+ mutex_unlock(&damon_lock);
+ return -EBUSY;
+ }
+
+ for (i = 0; i < nr_ctxs; i++) {
+ err = __damon_start(ctxs[i]);
+ if (err)
+ break;
+ nr_running_ctxs++;
+ }
+ mutex_unlock(&damon_lock);
+
+ return err;
+}
+
+/*
+ * __damon_stop() - Stops monitoring of given context.
+ * @ctx: monitoring context
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+static int __damon_stop(struct damon_ctx *ctx)
+{
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ ctx->kdamond_stop = true;
+ mutex_unlock(&ctx->kdamond_lock);
+ while (damon_kdamond_running(ctx))
+ usleep_range(ctx->sample_interval,
+ ctx->sample_interval * 2);
+ return 0;
+ }
+ mutex_unlock(&ctx->kdamond_lock);
+
+ return -EPERM;
+}
+
+/**
+ * damon_stop() - Stops the monitorings for a given group of contexts.
+ * @ctxs: an array of the pointers for contexts to stop monitoring
+ * @nr_ctxs: size of @ctxs
+ *
+ * Return: 0 on success, negative error code otherwise.
+ */
+int damon_stop(struct damon_ctx **ctxs, int nr_ctxs)
+{
+ int i, err = 0;
+
+ for (i = 0; i < nr_ctxs; i++) {
+ /* nr_running_ctxs is decremented in kdamond_fn */
+ err = __damon_stop(ctxs[i]);
+ if (err)
+ return err;
+ }
+
+ return err;
+}
+
+/*
+ * damon_check_reset_time_interval() - Check if a time interval is elapsed.
+ * @baseline: the time to check whether the interval has elapsed since
+ * @interval: the time interval (microseconds)
+ *
+ * See whether the given time interval has passed since the given baseline
+ * time. If so, it also updates the baseline to current time for next check.
+ *
+ * Return: true if the time interval has passed, or false otherwise.
+ */
+static bool damon_check_reset_time_interval(struct timespec64 *baseline,
+ unsigned long interval)
+{
+ struct timespec64 now;
+
+ ktime_get_coarse_ts64(&now);
+ if ((timespec64_to_ns(&now) - timespec64_to_ns(baseline)) <
+ interval * 1000)
+ return false;
+ *baseline = now;
+ return true;
+}
+
+/*
+ * Check whether it is time to flush the aggregated information
+ */
+static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx)
+{
+ return damon_check_reset_time_interval(&ctx->last_aggregation,
+ ctx->aggr_interval);
+}
+
+/*
+ * Check whether it is time to check and apply the target monitoring regions
+ *
+ * Returns true if it is.
+ */
+static bool kdamond_need_update_regions(struct damon_ctx *ctx)
+{
+ return damon_check_reset_time_interval(&ctx->last_regions_update,
+ ctx->regions_update_interval);
+}
+
+/*
+ * Check whether current monitoring should be stopped
+ *
+ * The monitoring is stopped when either the user requested to stop, or all
+ * monitoring targets are invalid.
+ *
+ * Returns true if need to stop current monitoring.
+ */
+static bool kdamond_need_stop(struct damon_ctx *ctx)
+{
+ bool stop;
+
+ mutex_lock(&ctx->kdamond_lock);
+ stop = ctx->kdamond_stop;
+ mutex_unlock(&ctx->kdamond_lock);
+ if (stop)
+ return true;
+
+ if (!ctx->primitive.target_valid)
+ return false;
+
+ return !ctx->primitive.target_valid(ctx->target);
+}
+
+static void set_kdamond_stop(struct damon_ctx *ctx)
+{
+ mutex_lock(&ctx->kdamond_lock);
+ ctx->kdamond_stop = true;
+ mutex_unlock(&ctx->kdamond_lock);
+}
+
+/*
+ * The monitoring daemon that runs as a kernel thread
+ */
+static int kdamond_fn(void *data)
+{
+ struct damon_ctx *ctx = (struct damon_ctx *)data;
+
+ pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
+
+ if (ctx->primitive.init_target_regions)
+ ctx->primitive.init_target_regions(ctx);
+ if (ctx->callback.before_start && ctx->callback.before_start(ctx))
+ set_kdamond_stop(ctx);
+
+ while (!kdamond_need_stop(ctx)) {
+ if (ctx->primitive.prepare_access_checks)
+ ctx->primitive.prepare_access_checks(ctx);
+ if (ctx->callback.after_sampling &&
+ ctx->callback.after_sampling(ctx))
+ set_kdamond_stop(ctx);
+
+ usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
+
+ if (ctx->primitive.check_accesses)
+ ctx->primitive.check_accesses(ctx);
+
+ if (kdamond_aggregate_interval_passed(ctx)) {
+ if (ctx->callback.after_aggregation &&
+ ctx->callback.after_aggregation(ctx))
+ set_kdamond_stop(ctx);
+ if (ctx->primitive.reset_aggregated)
+ ctx->primitive.reset_aggregated(ctx);
+ }
+
+ if (kdamond_need_update_regions(ctx)) {
+ if (ctx->primitive.update_target_regions)
+ ctx->primitive.update_target_regions(ctx);
+ }
+ }
+
+ if (ctx->callback.before_terminate &&
+ ctx->callback.before_terminate(ctx))
+ set_kdamond_stop(ctx);
+ if (ctx->primitive.cleanup)
+ ctx->primitive.cleanup(ctx);
+
+ pr_debug("kdamond (%d) finishes\n", ctx->kdamond->pid);
+ mutex_lock(&ctx->kdamond_lock);
+ ctx->kdamond = NULL;
+ mutex_unlock(&ctx->kdamond_lock);
+
+ mutex_lock(&damon_lock);
+ nr_running_ctxs--;
+ mutex_unlock(&damon_lock);
+
+ do_exit(0);
+}
--
2.17.1
From: SeongJae Park <[email protected]>
This commit adds kunit based unit tests for the core and the virtual
address spaces monitoring primitives of DAMON.
Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Brendan Higgins <[email protected]>
---
mm/damon/Kconfig | 36 +++++
mm/damon/core-test.h | 253 ++++++++++++++++++++++++++++++++
mm/damon/core.c | 7 +
mm/damon/dbgfs-test.h | 209 +++++++++++++++++++++++++++
mm/damon/dbgfs.c | 2 +
mm/damon/vaddr-test.h | 328 ++++++++++++++++++++++++++++++++++++++++++
mm/damon/vaddr.c | 7 +
7 files changed, 842 insertions(+)
create mode 100644 mm/damon/core-test.h
create mode 100644 mm/damon/dbgfs-test.h
create mode 100644 mm/damon/vaddr-test.h
diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
index 72f1683ba0ee..455995152697 100644
--- a/mm/damon/Kconfig
+++ b/mm/damon/Kconfig
@@ -12,6 +12,18 @@ config DAMON
See https://damonitor.github.io/doc/html/latest-damon/index.html for
more information.
+config DAMON_KUNIT_TEST
+ bool "Test for damon" if !KUNIT_ALL_TESTS
+ depends on DAMON && KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ This builds the DAMON Kunit test suite.
+
+ For more information on KUnit and unit tests in general, please refer
+ to the KUnit documentation.
+
+ If unsure, say N.
+
config DAMON_VADDR
bool "Data access monitoring primitives for virtual address spaces"
depends on DAMON && MMU
@@ -21,6 +33,18 @@ config DAMON_VADDR
This builds the default data access monitoring primitives for DAMON
that works for virtual address spaces.
+config DAMON_VADDR_KUNIT_TEST
+ bool "Test for DAMON primitives" if !KUNIT_ALL_TESTS
+ depends on DAMON_VADDR && KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ This builds the DAMON virtual addresses primitives Kunit test suite.
+
+ For more information on KUnit and unit tests in general, please refer
+ to the KUnit documentation.
+
+ If unsure, say N.
+
config DAMON_DBGFS
bool "DAMON debugfs interface"
depends on DAMON_VADDR && DEBUG_FS
@@ -30,4 +54,16 @@ config DAMON_DBGFS
If unsure, say N.
+config DAMON_DBGFS_KUNIT_TEST
+ bool "Test for damon debugfs interface" if !KUNIT_ALL_TESTS
+ depends on DAMON_DBGFS && KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ This builds the DAMON debugfs interface Kunit test suite.
+
+ For more information on KUnit and unit tests in general, please refer
+ to the KUnit documentation.
+
+ If unsure, say N.
+
endmenu
diff --git a/mm/damon/core-test.h b/mm/damon/core-test.h
new file mode 100644
index 000000000000..c1a5dcab6c07
--- /dev/null
+++ b/mm/damon/core-test.h
@@ -0,0 +1,253 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Data Access Monitor Unit Tests
+ *
+ * Copyright 2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#ifdef CONFIG_DAMON_KUNIT_TEST
+
+#ifndef _DAMON_CORE_TEST_H
+#define _DAMON_CORE_TEST_H
+
+#include <kunit/test.h>
+
+static void damon_test_regions(struct kunit *test)
+{
+ struct damon_region *r;
+ struct damon_target *t;
+
+ r = damon_new_region(1, 2);
+ KUNIT_EXPECT_EQ(test, 1ul, r->ar.start);
+ KUNIT_EXPECT_EQ(test, 2ul, r->ar.end);
+ KUNIT_EXPECT_EQ(test, 0u, r->nr_accesses);
+
+ t = damon_new_target(42);
+ KUNIT_EXPECT_EQ(test, 0u, damon_nr_regions(t));
+
+ damon_add_region(r, t);
+ KUNIT_EXPECT_EQ(test, 1u, damon_nr_regions(t));
+
+ damon_del_region(r);
+ KUNIT_EXPECT_EQ(test, 0u, damon_nr_regions(t));
+
+ damon_free_target(t);
+}
+
+static unsigned int nr_damon_targets(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ unsigned int nr_targets = 0;
+
+ damon_for_each_target(t, ctx)
+ nr_targets++;
+
+ return nr_targets;
+}
+
+static void damon_test_target(struct kunit *test)
+{
+ struct damon_ctx *c = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ struct damon_target *t;
+
+ t = damon_new_target(42);
+ KUNIT_EXPECT_EQ(test, 42ul, t->id);
+ KUNIT_EXPECT_EQ(test, 0u, nr_damon_targets(c));
+
+ damon_add_target(c, t);
+ KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(c));
+
+ damon_destroy_target(t);
+ KUNIT_EXPECT_EQ(test, 0u, nr_damon_targets(c));
+
+ damon_destroy_ctx(c);
+}
+
+/*
+ * Test kdamond_reset_aggregated()
+ *
+ * DAMON checks access to each region and aggregates this information as the
+ * access frequency of each region. In detail, it increases '->nr_accesses' of
+ * regions that an access has confirmed. 'kdamond_reset_aggregated()' flushes
+ * the aggregated information ('->nr_accesses' of each regions) to the result
+ * buffer. As a result of the flushing, the '->nr_accesses' of regions are
+ * initialized to zero.
+ */
+static void damon_test_aggregate(struct kunit *test)
+{
+ struct damon_ctx *ctx = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ unsigned long target_ids[] = {1, 2, 3};
+ unsigned long saddr[][3] = {{10, 20, 30}, {5, 42, 49}, {13, 33, 55} };
+ unsigned long eaddr[][3] = {{15, 27, 40}, {31, 45, 55}, {23, 44, 66} };
+ unsigned long accesses[][3] = {{42, 95, 84}, {10, 20, 30}, {0, 1, 2} };
+ struct damon_target *t;
+ struct damon_region *r;
+ int it, ir;
+
+ damon_set_targets(ctx, target_ids, 3);
+
+ it = 0;
+ damon_for_each_target(t, ctx) {
+ for (ir = 0; ir < 3; ir++) {
+ r = damon_new_region(saddr[it][ir], eaddr[it][ir]);
+ r->nr_accesses = accesses[it][ir];
+ damon_add_region(r, t);
+ }
+ it++;
+ }
+ kdamond_reset_aggregated(ctx);
+ it = 0;
+ damon_for_each_target(t, ctx) {
+ ir = 0;
+ /* '->nr_accesses' should be zeroed */
+ damon_for_each_region(r, t) {
+ KUNIT_EXPECT_EQ(test, 0u, r->nr_accesses);
+ ir++;
+ }
+ /* regions should be preserved */
+ KUNIT_EXPECT_EQ(test, 3, ir);
+ it++;
+ }
+ /* targets also should be preserved */
+ KUNIT_EXPECT_EQ(test, 3, it);
+
+ damon_destroy_ctx(ctx);
+}
+
+static void damon_test_split_at(struct kunit *test)
+{
+ struct damon_ctx *c = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ struct damon_target *t;
+ struct damon_region *r;
+
+ t = damon_new_target(42);
+ r = damon_new_region(0, 100);
+ damon_add_region(r, t);
+ damon_split_region_at(c, r, 25);
+ KUNIT_EXPECT_EQ(test, r->ar.start, 0ul);
+ KUNIT_EXPECT_EQ(test, r->ar.end, 25ul);
+
+ r = damon_next_region(r);
+ KUNIT_EXPECT_EQ(test, r->ar.start, 25ul);
+ KUNIT_EXPECT_EQ(test, r->ar.end, 100ul);
+
+ damon_free_target(t);
+ damon_destroy_ctx(c);
+}
+
+static void damon_test_merge_two(struct kunit *test)
+{
+ struct damon_target *t;
+ struct damon_region *r, *r2, *r3;
+ int i;
+
+ t = damon_new_target(42);
+ r = damon_new_region(0, 100);
+ r->nr_accesses = 10;
+ damon_add_region(r, t);
+ r2 = damon_new_region(100, 300);
+ r2->nr_accesses = 20;
+ damon_add_region(r2, t);
+
+ damon_merge_two_regions(r, r2);
+ KUNIT_EXPECT_EQ(test, r->ar.start, 0ul);
+ KUNIT_EXPECT_EQ(test, r->ar.end, 300ul);
+ KUNIT_EXPECT_EQ(test, r->nr_accesses, 16u);
+
+ i = 0;
+ damon_for_each_region(r3, t) {
+ KUNIT_EXPECT_PTR_EQ(test, r, r3);
+ i++;
+ }
+ KUNIT_EXPECT_EQ(test, i, 1);
+
+ damon_free_target(t);
+}
+
+static struct damon_region *__nth_region_of(struct damon_target *t, int idx)
+{
+ struct damon_region *r;
+ unsigned int i = 0;
+
+ damon_for_each_region(r, t) {
+ if (i++ == idx)
+ return r;
+ }
+
+ return NULL;
+}
+
+static void damon_test_merge_regions_of(struct kunit *test)
+{
+ struct damon_target *t;
+ struct damon_region *r;
+ unsigned long sa[] = {0, 100, 114, 122, 130, 156, 170, 184};
+ unsigned long ea[] = {100, 112, 122, 130, 156, 170, 184, 230};
+ unsigned int nrs[] = {0, 0, 10, 10, 20, 30, 1, 2};
+
+ unsigned long saddrs[] = {0, 114, 130, 156, 170};
+ unsigned long eaddrs[] = {112, 130, 156, 170, 230};
+ int i;
+
+ t = damon_new_target(42);
+ for (i = 0; i < ARRAY_SIZE(sa); i++) {
+ r = damon_new_region(sa[i], ea[i]);
+ r->nr_accesses = nrs[i];
+ damon_add_region(r, t);
+ }
+
+ damon_merge_regions_of(t, 9, 9999);
+ /* 0-112, 114-130, 130-156, 156-170 */
+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 5u);
+ for (i = 0; i < 5; i++) {
+ r = __nth_region_of(t, i);
+ KUNIT_EXPECT_EQ(test, r->ar.start, saddrs[i]);
+ KUNIT_EXPECT_EQ(test, r->ar.end, eaddrs[i]);
+ }
+ damon_free_target(t);
+}
+
+static void damon_test_split_regions_of(struct kunit *test)
+{
+ struct damon_ctx *c = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ struct damon_target *t;
+ struct damon_region *r;
+
+ t = damon_new_target(42);
+ r = damon_new_region(0, 22);
+ damon_add_region(r, t);
+ damon_split_regions_of(c, t, 2);
+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 2u);
+ damon_free_target(t);
+
+ t = damon_new_target(42);
+ r = damon_new_region(0, 220);
+ damon_add_region(r, t);
+ damon_split_regions_of(c, t, 4);
+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 4u);
+ damon_free_target(t);
+ damon_destroy_ctx(c);
+}
+
+static struct kunit_case damon_test_cases[] = {
+ KUNIT_CASE(damon_test_target),
+ KUNIT_CASE(damon_test_regions),
+ KUNIT_CASE(damon_test_aggregate),
+ KUNIT_CASE(damon_test_split_at),
+ KUNIT_CASE(damon_test_merge_two),
+ KUNIT_CASE(damon_test_merge_regions_of),
+ KUNIT_CASE(damon_test_split_regions_of),
+ {},
+};
+
+static struct kunit_suite damon_test_suite = {
+ .name = "damon",
+ .test_cases = damon_test_cases,
+};
+kunit_test_suite(damon_test_suite);
+
+#endif /* _DAMON_CORE_TEST_H */
+
+#endif /* CONFIG_DAMON_KUNIT_TEST */
diff --git a/mm/damon/core.c b/mm/damon/core.c
index b9575a6bebff..214ff404a423 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -16,6 +16,11 @@
#define CREATE_TRACE_POINTS
#include <trace/events/damon.h>
+#ifdef CONFIG_DAMON_KUNIT_TEST
+#undef DAMON_MIN_REGION
+#define DAMON_MIN_REGION 1
+#endif
+
/* Get a random number in [l, r) */
#define damon_rand(l, r) (l + prandom_u32_max(r - l))
@@ -708,3 +713,5 @@ static int kdamond_fn(void *data)
do_exit(0);
}
+
+#include "core-test.h"
diff --git a/mm/damon/dbgfs-test.h b/mm/damon/dbgfs-test.h
new file mode 100644
index 000000000000..ce9c6784ad47
--- /dev/null
+++ b/mm/damon/dbgfs-test.h
@@ -0,0 +1,209 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * DAMON Debugfs Interface Unit Tests
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#ifdef CONFIG_DAMON_DBGFS_KUNIT_TEST
+
+#ifndef _DAMON_DBGFS_TEST_H
+#define _DAMON_DBGFS_TEST_H
+
+#include <kunit/test.h>
+
+static void damon_dbgfs_test_str_to_target_ids(struct kunit *test)
+{
+ char *question;
+ unsigned long *answers;
+ unsigned long expected[] = {12, 35, 46};
+ ssize_t nr_integers = 0, i;
+
+ question = "123";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)1, nr_integers);
+ KUNIT_EXPECT_EQ(test, 123ul, answers[0]);
+ kfree(answers);
+
+ question = "123abc";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)1, nr_integers);
+ KUNIT_EXPECT_EQ(test, 123ul, answers[0]);
+ kfree(answers);
+
+ question = "a123";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)0, nr_integers);
+ kfree(answers);
+
+ question = "12 35";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)2, nr_integers);
+ for (i = 0; i < nr_integers; i++)
+ KUNIT_EXPECT_EQ(test, expected[i], answers[i]);
+ kfree(answers);
+
+ question = "12 35 46";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)3, nr_integers);
+ for (i = 0; i < nr_integers; i++)
+ KUNIT_EXPECT_EQ(test, expected[i], answers[i]);
+ kfree(answers);
+
+ question = "12 35 abc 46";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)2, nr_integers);
+ for (i = 0; i < 2; i++)
+ KUNIT_EXPECT_EQ(test, expected[i], answers[i]);
+ kfree(answers);
+
+ question = "";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)0, nr_integers);
+ kfree(answers);
+
+ question = "\n";
+ answers = str_to_target_ids(question, strnlen(question, 128),
+ &nr_integers);
+ KUNIT_EXPECT_EQ(test, (ssize_t)0, nr_integers);
+ kfree(answers);
+}
+
+static void damon_dbgfs_test_set_targets(struct kunit *test)
+{
+ struct damon_ctx *ctx = dbgfs_new_ctx();
+ unsigned long ids[] = {1, 2, 3};
+ char buf[64];
+
+ /* Make DAMON consider target id as plain number */
+ ctx->primitive.target_valid = NULL;
+
+ damon_set_targets(ctx, ids, 3);
+ sprint_target_ids(ctx, buf, 64);
+ KUNIT_EXPECT_STREQ(test, (char *)buf, "1 2 3\n");
+
+ damon_set_targets(ctx, NULL, 0);
+ sprint_target_ids(ctx, buf, 64);
+ KUNIT_EXPECT_STREQ(test, (char *)buf, "\n");
+
+ damon_set_targets(ctx, (unsigned long []){1, 2}, 2);
+ sprint_target_ids(ctx, buf, 64);
+ KUNIT_EXPECT_STREQ(test, (char *)buf, "1 2\n");
+
+ damon_set_targets(ctx, (unsigned long []){2}, 1);
+ sprint_target_ids(ctx, buf, 64);
+ KUNIT_EXPECT_STREQ(test, (char *)buf, "2\n");
+
+ damon_set_targets(ctx, NULL, 0);
+ sprint_target_ids(ctx, buf, 64);
+ KUNIT_EXPECT_STREQ(test, (char *)buf, "\n");
+
+ dbgfs_destroy_ctx(ctx);
+}
+
+static void damon_dbgfs_test_set_recording(struct kunit *test)
+{
+ struct damon_ctx *ctx = dbgfs_new_ctx();
+ struct dbgfs_recorder *rec = ctx->callback.private;
+ int err;
+
+ err = dbgfs_set_recording(ctx, 42, "foo");
+ KUNIT_EXPECT_EQ(test, err, -EINVAL);
+ dbgfs_set_recording(ctx, 4242, "foo.bar");
+ KUNIT_EXPECT_EQ(test, rec->rbuf_len, 4242u);
+ KUNIT_EXPECT_STREQ(test, rec->rfile_path, "foo.bar");
+ dbgfs_set_recording(ctx, 424242, "foo");
+ KUNIT_EXPECT_EQ(test, rec->rbuf_len, 424242u);
+ KUNIT_EXPECT_STREQ(test, rec->rfile_path, "foo");
+
+ dbgfs_destroy_ctx(ctx);
+}
+
+static void damon_dbgfs_test_write_rbuf(struct kunit *test)
+{
+ struct damon_ctx *ctx = dbgfs_new_ctx();
+ struct dbgfs_recorder *rec = ctx->callback.private;
+ char *data;
+
+ dbgfs_set_recording(ctx, 4242, "damon.data");
+
+ data = "hello";
+ dbgfs_write_rbuf(ctx, data, strnlen(data, 256));
+ KUNIT_EXPECT_EQ(test, rec->rbuf_offset, 5u);
+
+ dbgfs_write_rbuf(ctx, data, 0);
+ KUNIT_EXPECT_EQ(test, rec->rbuf_offset, 5u);
+
+ KUNIT_EXPECT_STREQ(test, (char *)rec->rbuf, data);
+
+ dbgfs_destroy_ctx(ctx);
+}
+
+/*
+ * Test dbgfs_after_aggregation()
+ *
+ * dbgfs sets dbgfs_after_aggregation() as aggregate callback. It stores the
+ * aggregated monitoring information ('->nr_accesses' of each regions) to the
+ * result buffer.
+ */
+static void damon_dbgfs_test_aggregate(struct kunit *test)
+{
+ struct damon_ctx *ctx = dbgfs_new_ctx();
+ struct dbgfs_recorder *rec = ctx->callback.private;
+ unsigned long target_ids[] = {1, 2, 3};
+ unsigned long saddr[][3] = {{10, 20, 30}, {5, 42, 49}, {13, 33, 55} };
+ unsigned long eaddr[][3] = {{15, 27, 40}, {31, 45, 55}, {23, 44, 66} };
+ unsigned long accesses[][3] = {{42, 95, 84}, {10, 20, 30}, {0, 1, 2} };
+ struct damon_target *t;
+ struct damon_region *r;
+ int it, ir;
+ ssize_t sz, sr, sp;
+
+ dbgfs_set_recording(ctx, 4242, "damon.data");
+ damon_set_targets(ctx, target_ids, 3);
+
+ it = 0;
+ damon_for_each_target(t, ctx) {
+ for (ir = 0; ir < 3; ir++) {
+ r = damon_new_region(saddr[it][ir], eaddr[it][ir]);
+ r->nr_accesses = accesses[it][ir];
+ damon_add_region(r, t);
+ }
+ it++;
+ }
+ dbgfs_after_aggregation(ctx);
+
+ /* The aggregated information should be written in the buffer */
+ sr = sizeof(r->ar.start) + sizeof(r->ar.end) + sizeof(r->nr_accesses);
+ sp = sizeof(t->id) + sizeof(unsigned int) + 3 * sr;
+ sz = sizeof(struct timespec64) + sizeof(unsigned int) + 3 * sp;
+ KUNIT_EXPECT_EQ(test, (unsigned int)sz, rec->rbuf_offset);
+
+ damon_destroy_ctx(ctx);
+}
+
+static struct kunit_case damon_test_cases[] = {
+ KUNIT_CASE(damon_dbgfs_test_str_to_target_ids),
+ KUNIT_CASE(damon_dbgfs_test_set_targets),
+ KUNIT_CASE(damon_dbgfs_test_set_recording),
+ KUNIT_CASE(damon_dbgfs_test_write_rbuf),
+ KUNIT_CASE(damon_dbgfs_test_aggregate),
+ {},
+};
+
+static struct kunit_suite damon_test_suite = {
+ .name = "damon-dbgfs",
+ .test_cases = damon_test_cases,
+};
+kunit_test_suite(damon_test_suite);
+
+#endif /* _DAMON_TEST_H */
+
+#endif /* CONFIG_DAMON_KUNIT_TEST */
diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
index 46186b057e1a..7380c6502f3d 100644
--- a/mm/damon/dbgfs.c
+++ b/mm/damon/dbgfs.c
@@ -863,3 +863,5 @@ static int __init damon_dbgfs_init(void)
}
module_init(damon_dbgfs_init);
+
+#include "dbgfs-test.h"
diff --git a/mm/damon/vaddr-test.h b/mm/damon/vaddr-test.h
new file mode 100644
index 000000000000..bd36161d00f4
--- /dev/null
+++ b/mm/damon/vaddr-test.h
@@ -0,0 +1,328 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Data Access Monitor Unit Tests
+ *
+ * Copyright 2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ *
+ * Author: SeongJae Park <[email protected]>
+ */
+
+#ifdef CONFIG_DAMON_VADDR_KUNIT_TEST
+
+#ifndef _DAMON_VADDR_TEST_H
+#define _DAMON_VADDR_TEST_H
+
+#include <kunit/test.h>
+
+static void __link_vmas(struct vm_area_struct *vmas, ssize_t nr_vmas)
+{
+ int i, j;
+ unsigned long largest_gap, gap;
+
+ if (!nr_vmas)
+ return;
+
+ for (i = 0; i < nr_vmas - 1; i++) {
+ vmas[i].vm_next = &vmas[i + 1];
+
+ vmas[i].vm_rb.rb_left = NULL;
+ vmas[i].vm_rb.rb_right = &vmas[i + 1].vm_rb;
+
+ largest_gap = 0;
+ for (j = i; j < nr_vmas; j++) {
+ if (j == 0)
+ continue;
+ gap = vmas[j].vm_start - vmas[j - 1].vm_end;
+ if (gap > largest_gap)
+ largest_gap = gap;
+ }
+ vmas[i].rb_subtree_gap = largest_gap;
+ }
+ vmas[i].vm_next = NULL;
+ vmas[i].vm_rb.rb_right = NULL;
+ vmas[i].rb_subtree_gap = 0;
+}
+
+/*
+ * Test __damon_va_three_regions() function
+ *
+ * In case of virtual memory address spaces monitoring, DAMON converts the
+ * complex and dynamic memory mappings of each target task to three
+ * discontiguous regions which cover every mapped areas. However, the three
+ * regions should not include the two biggest unmapped areas in the original
+ * mapping, because the two biggest areas are normally the areas between 1)
+ * heap and the mmap()-ed regions, and 2) the mmap()-ed regions and stack.
+ * Because these two unmapped areas are very huge but obviously never accessed,
+ * covering the region is just a waste.
+ *
+ * '__damon_va_three_regions() receives an address space of a process. It
+ * first identifies the start of mappings, end of mappings, and the two biggest
+ * unmapped areas. After that, based on the information, it constructs the
+ * three regions and returns. For more detail, refer to the comment of
+ * 'damon_init_regions_of()' function definition in 'mm/damon.c' file.
+ *
+ * For example, suppose virtual address ranges of 10-20, 20-25, 200-210,
+ * 210-220, 300-305, and 307-330 (Other comments represent this mappings in
+ * more short form: 10-20-25, 200-210-220, 300-305, 307-330) of a process are
+ * mapped. To cover every mappings, the three regions should start with 10,
+ * and end with 305. The process also has three unmapped areas, 25-200,
+ * 220-300, and 305-307. Among those, 25-200 and 220-300 are the biggest two
+ * unmapped areas, and thus it should be converted to three regions of 10-25,
+ * 200-220, and 300-330.
+ */
+static void damon_test_three_regions_in_vmas(struct kunit *test)
+{
+ struct damon_addr_range regions[3] = {0,};
+ /* 10-20-25, 200-210-220, 300-305, 307-330 */
+ struct vm_area_struct vmas[] = {
+ (struct vm_area_struct) {.vm_start = 10, .vm_end = 20},
+ (struct vm_area_struct) {.vm_start = 20, .vm_end = 25},
+ (struct vm_area_struct) {.vm_start = 200, .vm_end = 210},
+ (struct vm_area_struct) {.vm_start = 210, .vm_end = 220},
+ (struct vm_area_struct) {.vm_start = 300, .vm_end = 305},
+ (struct vm_area_struct) {.vm_start = 307, .vm_end = 330},
+ };
+
+ __link_vmas(vmas, 6);
+
+ __damon_va_three_regions(&vmas[0], regions);
+
+ KUNIT_EXPECT_EQ(test, 10ul, regions[0].start);
+ KUNIT_EXPECT_EQ(test, 25ul, regions[0].end);
+ KUNIT_EXPECT_EQ(test, 200ul, regions[1].start);
+ KUNIT_EXPECT_EQ(test, 220ul, regions[1].end);
+ KUNIT_EXPECT_EQ(test, 300ul, regions[2].start);
+ KUNIT_EXPECT_EQ(test, 330ul, regions[2].end);
+}
+
+static struct damon_region *__nth_region_of(struct damon_target *t, int idx)
+{
+ struct damon_region *r;
+ unsigned int i = 0;
+
+ damon_for_each_region(r, t) {
+ if (i++ == idx)
+ return r;
+ }
+
+ return NULL;
+}
+
+/*
+ * Test 'damon_va_apply_three_regions()'
+ *
+ * test kunit object
+ * regions an array containing start/end addresses of current
+ * monitoring target regions
+ * nr_regions the number of the addresses in 'regions'
+ * three_regions The three regions that need to be applied now
+ * expected start/end addresses of monitoring target regions that
+ * 'three_regions' are applied
+ * nr_expected the number of addresses in 'expected'
+ *
+ * The memory mapping of the target processes changes dynamically. To follow
+ * the change, DAMON periodically reads the mappings, simplifies it to the
+ * three regions, and updates the monitoring target regions to fit in the three
+ * regions. The update of current target regions is the role of
+ * 'damon_va_apply_three_regions()'.
+ *
+ * This test passes the given target regions and the new three regions that
+ * need to be applied to the function and check whether it updates the regions
+ * as expected.
+ */
+static void damon_do_test_apply_three_regions(struct kunit *test,
+ unsigned long *regions, int nr_regions,
+ struct damon_addr_range *three_regions,
+ unsigned long *expected, int nr_expected)
+{
+ struct damon_ctx *ctx = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ struct damon_target *t;
+ struct damon_region *r;
+ int i;
+
+ t = damon_new_target(42);
+ for (i = 0; i < nr_regions / 2; i++) {
+ r = damon_new_region(regions[i * 2], regions[i * 2 + 1]);
+ damon_add_region(r, t);
+ }
+ damon_add_target(ctx, t);
+
+ damon_va_apply_three_regions(ctx, t, three_regions);
+
+ for (i = 0; i < nr_expected / 2; i++) {
+ r = __nth_region_of(t, i);
+ KUNIT_EXPECT_EQ(test, r->ar.start, expected[i * 2]);
+ KUNIT_EXPECT_EQ(test, r->ar.end, expected[i * 2 + 1]);
+ }
+
+ damon_destroy_ctx(ctx);
+}
+
+/*
+ * This function test most common case where the three big regions are only
+ * slightly changed. Target regions should adjust their boundary (10-20-30,
+ * 50-55, 70-80, 90-100) to fit with the new big regions or remove target
+ * regions (57-79) that now out of the three regions.
+ */
+static void damon_test_apply_three_regions1(struct kunit *test)
+{
+ /* 10-20-30, 50-55-57-59, 70-80-90-100 */
+ unsigned long regions[] = {10, 20, 20, 30, 50, 55, 55, 57, 57, 59,
+ 70, 80, 80, 90, 90, 100};
+ /* 5-27, 45-55, 73-104 */
+ struct damon_addr_range new_three_regions[3] = {
+ (struct damon_addr_range){.start = 5, .end = 27},
+ (struct damon_addr_range){.start = 45, .end = 55},
+ (struct damon_addr_range){.start = 73, .end = 104} };
+ /* 5-20-27, 45-55, 73-80-90-104 */
+ unsigned long expected[] = {5, 20, 20, 27, 45, 55,
+ 73, 80, 80, 90, 90, 104};
+
+ damon_do_test_apply_three_regions(test, regions, ARRAY_SIZE(regions),
+ new_three_regions, expected, ARRAY_SIZE(expected));
+}
+
+/*
+ * Test slightly bigger change. Similar to above, but the second big region
+ * now require two target regions (50-55, 57-59) to be removed.
+ */
+static void damon_test_apply_three_regions2(struct kunit *test)
+{
+ /* 10-20-30, 50-55-57-59, 70-80-90-100 */
+ unsigned long regions[] = {10, 20, 20, 30, 50, 55, 55, 57, 57, 59,
+ 70, 80, 80, 90, 90, 100};
+ /* 5-27, 56-57, 65-104 */
+ struct damon_addr_range new_three_regions[3] = {
+ (struct damon_addr_range){.start = 5, .end = 27},
+ (struct damon_addr_range){.start = 56, .end = 57},
+ (struct damon_addr_range){.start = 65, .end = 104} };
+ /* 5-20-27, 56-57, 65-80-90-104 */
+ unsigned long expected[] = {5, 20, 20, 27, 56, 57,
+ 65, 80, 80, 90, 90, 104};
+
+ damon_do_test_apply_three_regions(test, regions, ARRAY_SIZE(regions),
+ new_three_regions, expected, ARRAY_SIZE(expected));
+}
+
+/*
+ * Test a big change. The second big region has totally freed and mapped to
+ * different area (50-59 -> 61-63). The target regions which were in the old
+ * second big region (50-55-57-59) should be removed and new target region
+ * covering the second big region (61-63) should be created.
+ */
+static void damon_test_apply_three_regions3(struct kunit *test)
+{
+ /* 10-20-30, 50-55-57-59, 70-80-90-100 */
+ unsigned long regions[] = {10, 20, 20, 30, 50, 55, 55, 57, 57, 59,
+ 70, 80, 80, 90, 90, 100};
+ /* 5-27, 61-63, 65-104 */
+ struct damon_addr_range new_three_regions[3] = {
+ (struct damon_addr_range){.start = 5, .end = 27},
+ (struct damon_addr_range){.start = 61, .end = 63},
+ (struct damon_addr_range){.start = 65, .end = 104} };
+ /* 5-20-27, 61-63, 65-80-90-104 */
+ unsigned long expected[] = {5, 20, 20, 27, 61, 63,
+ 65, 80, 80, 90, 90, 104};
+
+ damon_do_test_apply_three_regions(test, regions, ARRAY_SIZE(regions),
+ new_three_regions, expected, ARRAY_SIZE(expected));
+}
+
+/*
+ * Test another big change. Both of the second and third big regions (50-59
+ * and 70-100) has totally freed and mapped to different area (30-32 and
+ * 65-68). The target regions which were in the old second and third big
+ * regions should now be removed and new target regions covering the new second
+ * and third big regions should be crated.
+ */
+static void damon_test_apply_three_regions4(struct kunit *test)
+{
+ /* 10-20-30, 50-55-57-59, 70-80-90-100 */
+ unsigned long regions[] = {10, 20, 20, 30, 50, 55, 55, 57, 57, 59,
+ 70, 80, 80, 90, 90, 100};
+ /* 5-7, 30-32, 65-68 */
+ struct damon_addr_range new_three_regions[3] = {
+ (struct damon_addr_range){.start = 5, .end = 7},
+ (struct damon_addr_range){.start = 30, .end = 32},
+ (struct damon_addr_range){.start = 65, .end = 68} };
+ /* expect 5-7, 30-32, 65-68 */
+ unsigned long expected[] = {5, 7, 30, 32, 65, 68};
+
+ damon_do_test_apply_three_regions(test, regions, ARRAY_SIZE(regions),
+ new_three_regions, expected, ARRAY_SIZE(expected));
+}
+
+static void damon_test_split_evenly(struct kunit *test)
+{
+ struct damon_ctx *c = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
+ struct damon_target *t;
+ struct damon_region *r;
+ unsigned long i;
+
+ KUNIT_EXPECT_EQ(test, damon_va_evenly_split_region(c, NULL, 5), -EINVAL);
+
+ t = damon_new_target(42);
+ r = damon_new_region(0, 100);
+ KUNIT_EXPECT_EQ(test, damon_va_evenly_split_region(c, r, 0), -EINVAL);
+
+ damon_add_region(r, t);
+ KUNIT_EXPECT_EQ(test, damon_va_evenly_split_region(c, r, 10), 0);
+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 10u);
+
+ i = 0;
+ damon_for_each_region(r, t) {
+ KUNIT_EXPECT_EQ(test, r->ar.start, i++ * 10);
+ KUNIT_EXPECT_EQ(test, r->ar.end, i * 10);
+ }
+ damon_free_target(t);
+
+ t = damon_new_target(42);
+ r = damon_new_region(5, 59);
+ damon_add_region(r, t);
+ KUNIT_EXPECT_EQ(test, damon_va_evenly_split_region(c, r, 5), 0);
+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 5u);
+
+ i = 0;
+ damon_for_each_region(r, t) {
+ if (i == 4)
+ break;
+ KUNIT_EXPECT_EQ(test, r->ar.start, 5 + 10 * i++);
+ KUNIT_EXPECT_EQ(test, r->ar.end, 5 + 10 * i);
+ }
+ KUNIT_EXPECT_EQ(test, r->ar.start, 5 + 10 * i);
+ KUNIT_EXPECT_EQ(test, r->ar.end, 59ul);
+ damon_free_target(t);
+
+ t = damon_new_target(42);
+ r = damon_new_region(5, 6);
+ damon_add_region(r, t);
+ KUNIT_EXPECT_EQ(test, damon_va_evenly_split_region(c, r, 2), -EINVAL);
+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 1u);
+
+ damon_for_each_region(r, t) {
+ KUNIT_EXPECT_EQ(test, r->ar.start, 5ul);
+ KUNIT_EXPECT_EQ(test, r->ar.end, 6ul);
+ }
+ damon_free_target(t);
+ damon_destroy_ctx(c);
+}
+
+static struct kunit_case damon_test_cases[] = {
+ KUNIT_CASE(damon_test_three_regions_in_vmas),
+ KUNIT_CASE(damon_test_apply_three_regions1),
+ KUNIT_CASE(damon_test_apply_three_regions2),
+ KUNIT_CASE(damon_test_apply_three_regions3),
+ KUNIT_CASE(damon_test_apply_three_regions4),
+ KUNIT_CASE(damon_test_split_evenly),
+ {},
+};
+
+static struct kunit_suite damon_test_suite = {
+ .name = "damon-primitives",
+ .test_cases = damon_test_cases,
+};
+kunit_test_suite(damon_test_suite);
+
+#endif /* _DAMON_VADDR_TEST_H */
+
+#endif /* CONFIG_DAMON_VADDR_KUNIT_TEST */
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index efc56ef843fe..8ee48a4abc17 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -15,6 +15,11 @@
#include <linux/sched/mm.h>
#include <linux/slab.h>
+#ifdef CONFIG_DAMON_VADDR_KUNIT_TEST
+#undef DAMON_MIN_REGION
+#define DAMON_MIN_REGION 1
+#endif
+
/* Get a random number in [l, r) */
#define damon_rand(l, r) (l + prandom_u32_max(r - l))
@@ -577,3 +582,5 @@ void damon_va_set_primitives(struct damon_ctx *ctx)
ctx->primitive.target_valid = damon_va_target_valid;
ctx->primitive.cleanup = damon_va_cleanup;
}
+
+#include "vaddr-test.h"
--
2.17.1
First I would like you to prune your To/CC list.
On Tue, Dec 15, 2020 at 3:56 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> DAMON is a data access monitoring framework for the Linux kernel. The
> core mechanisms of DAMON make it
>
> - accurate (the monitoring output is useful enough for DRAM level
> performance-centric memory management; It might be inappropriate for
> CPU cache levels, though),
> - light-weight (the monitoring overhead is normally low enough to be
> applied online), and
> - scalable (the upper-bound of the overhead is in constant range
> regardless of the size of target workloads).
>
> Using this framework, hence, we can easily write efficient kernel space
> data access monitoring applications. For example, the kernel's memory
> management mechanisms can make advanced decisions using this.
> Experimental data access aware optimization works that incurring high
> access monitoring overhead could implemented again on top of this.
*could again be implemented on*
>
> Due to its simple and flexible interface, providing user space interface
> would be also easy. Then, user space users who have some special
> workloads can write personalized applications for better understanding
> and optimizations of their workloads and systems.
>
> ---
>
> Nevertheless, this commit is defining and implementing only basic access
> check part without the overhead-accuracy handling core logic. The basic
> access check is as below.
>
> The output of DAMON says what memory regions are how frequently accessed
> for a given duration. The resolution of the access frequency is
> controlled by setting ``sampling interval`` and ``aggregation
> interval``. In detail, DAMON checks access to each page per ``sampling
> interval`` and aggregates the results. In other words, counts the
> number of the accesses to each region. After each ``aggregation
> interval`` passes, DAMON calls callback functions that previously
> registered by users so that users can read the aggregated results and
> then clears the results. This can be described in below simple
> pseudo-code::
>
> while monitoring_on:
> for page in monitoring_target:
> if accessed(page):
> nr_accesses[page] += 1
> if time() % aggregation_interval == 0:
> for callback in user_registered_callbacks:
> callback(monitoring_target, nr_accesses)
> for page in monitoring_target:
> nr_accesses[page] = 0
> sleep(sampling interval)
>
The above is a good example and I was hoping to see the almost same actual code.
> The target regions constructed at the beginning of the monitoring and
> updated after each ``regions_update_interval``, because the target
> regions could be dynamically changed (e.g., mmap() or memory hotplug).
> The monitoring overhead of this mechanism will arbitrarily increase as
> the size of the target workload grows.
>
> The basic monitoring primitives for actual access check and dynamic
> target regions construction aren't in the core part of DAMON. Instead,
> it allows users to implement their own primitives that optimized for
'that are optimized for'
> their use case and configure DAMON to use those. In other words, users
> cannot use current version of DAMON without some additional works.
>
> Following commits will implement the core mechanisms for the
> overhead-accuracy control and default primitives implementations.
>
> Signed-off-by: SeongJae Park <[email protected]>
> Reviewed-by: Leonard Foerster <[email protected]>
> ---
> include/linux/damon.h | 167 ++++++++++++++++++++++
> mm/Kconfig | 2 +
> mm/Makefile | 1 +
> mm/damon/Kconfig | 15 ++
> mm/damon/Makefile | 3 +
> mm/damon/core.c | 316 ++++++++++++++++++++++++++++++++++++++++++
> 6 files changed, 504 insertions(+)
> create mode 100644 include/linux/damon.h
> create mode 100644 mm/damon/Kconfig
> create mode 100644 mm/damon/Makefile
> create mode 100644 mm/damon/core.c
>
> diff --git a/include/linux/damon.h b/include/linux/damon.h
> new file mode 100644
> index 000000000000..387fa4399fc8
> --- /dev/null
> +++ b/include/linux/damon.h
> @@ -0,0 +1,167 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * DAMON api
> + *
> + * Author: SeongJae Park <[email protected]>
> + */
> +
> +#ifndef _DAMON_H_
> +#define _DAMON_H_
> +
> +#include <linux/mutex.h>
> +#include <linux/time64.h>
> +#include <linux/types.h>
> +
> +struct damon_ctx;
> +
> +/**
> + * struct damon_primitive Monitoring primitives for given use cases.
> + *
> + * @init_target_regions: Constructs initial monitoring target regions.
> + * @update_target_regions: Updates monitoring target regions.
The 'region' here still does not feel right. I prefer to rename the
above two to just init() and update().
> + * @prepare_access_checks: Prepares next access check of target regions.
> + * @check_accesses: Checks the access of target regions.
> + * @reset_aggregated: Resets aggregated accesses monitoring results.
> + * @target_valid: Determine if the target is valid.
> + * @cleanup: Cleans up the context.
> + *
> + * DAMON can be extended for various address spaces and usages. For this,
> + * users should register the low level primitives for their target address
> + * space and usecase via the &damon_ctx.primitive. Then, the monitoring thread
> + * calls @init_target_regions and @prepare_access_checks before starting the
> + * monitoring, @update_target_regions after each
> + * &damon_ctx.regions_update_interval, and @check_accesses, @target_valid and
> + * @prepare_access_checks after each &damon_ctx.sample_interval. Finally,
> + * @reset_aggregated is called after each &damon_ctx.aggr_interval.
> + *
> + * @init_target_regions should construct proper monitoring target regions and
> + * link those to the DAMON context struct. The regions should be defined by
> + * user and saved in @damon_ctx.target.
> + * @update_target_regions should update the monitoring target regions for
> + * current status.
> + * @prepare_access_checks should manipulate the monitoring regions to be
> + * prepared for the next access check.
> + * @check_accesses should check the accesses to each region that made after the
> + * last preparation and update the number of observed accesses of each region.
> + * @reset_aggregated should reset the access monitoring results that aggregated
> + * by @check_accesses.
> + * @target_valid should check whether the target is still valid for the
> + * monitoring.
> + * @cleanup is called from @kdamond just before its termination. After this
> + * call, only @kdamond_lock and @kdamond will be touched.
> + */
> +struct damon_primitive {
> + void (*init_target_regions)(struct damon_ctx *context);
> + void (*update_target_regions)(struct damon_ctx *context);
> + void (*prepare_access_checks)(struct damon_ctx *context);
> + void (*check_accesses)(struct damon_ctx *context);
> + void (*reset_aggregated)(struct damon_ctx *context);
> + bool (*target_valid)(void *target);
> + void (*cleanup)(struct damon_ctx *context);
> +};
> +
> +/*
> + * struct damon_callback Monitoring events notification callbacks.
> + *
> + * @before_start: Called before starting the monitoring.
> + * @after_sampling: Called after each sampling.
> + * @after_aggregation: Called after each aggregation.
> + * @before_terminate: Called before terminating the monitoring.
> + * @private: User private data.
> + *
> + * The monitoring thread (&damon_ctx->kdamond) calls @before_start and
> + * @before_terminate just before starting and finishing the monitoring,
> + * respectively. Therefore, those are good places for installing and cleaning
> + * @private.
> + *
> + * The monitoring thread calls @after_sampling and @after_aggregation for each
> + * of the sampling intervals and aggregation intervals, respectively.
> + * Therefore, users can safely access the monitoring results without additional
> + * protection. For the reason, users are recommended to use these callback for
> + * the accesses to the results.
> + *
> + * If any callback returns non-zero, monitoring stops.
I am not sure if this is the right patch to add this struct. Either we
need to add text why this is needed or add this patch when the real
user i.e. debugfs interface is added.
> + */
> +struct damon_callback {
> + void *private;
> +
> + int (*before_start)(struct damon_ctx *context);
> + int (*after_sampling)(struct damon_ctx *context);
> + int (*after_aggregation)(struct damon_ctx *context);
> + int (*before_terminate)(struct damon_ctx *context);
> +};
> +
> +/**
> + * struct damon_ctx - Represents a context for each monitoring. This is the
> + * main interface that allows users to set the attributes and get the results
> + * of the monitoring.
> + *
> + * @sample_interval: The time between access samplings.
> + * @aggr_interval: The time between monitor results aggregations.
> + * @regions_update_interval: The time between monitor regions updates.
regions_update_internal should be part of the primitive abstraction
and the update() callback internally can check this field.
> + *
> + * For each @sample_interval, DAMON checks whether each region is accessed or
> + * not. It aggregates and keeps the access information (number of accesses to
> + * each region) for @aggr_interval time. DAMON also checks whether the target
> + * memory regions need update (e.g., by ``mmap()`` calls from the application,
> + * in case of virtual memory monitoring) and applies the changes for each
> + * @regions_update_interval. All time intervals are in micro-seconds. Please
> + * refer to &struct damon_primitive and &struct damon_callback for more detail.
> + *
> + * @kdamond: Kernel thread who does the monitoring.
> + * @kdamond_stop: Notifies whether kdamond should stop.
> + * @kdamond_lock: Mutex for the synchronizations with @kdamond.
> + *
> + * For each monitoring context, one kernel thread for the monitoring is
> + * created. The pointer to the thread is stored in @kdamond.
> + *
> + * Once started, the monitoring thread runs until explicitly required to be
> + * terminated or every monitoring target is invalid. The validity of the
> + * targets is checked via the &damon_primitive.target_valid of @primitive. The
> + * termination can also be explicitly requested by writing non-zero to
> + * @kdamond_stop. The thread sets @kdamond to NULL when it terminates.
> + * Therefore, users can know whether the monitoring is ongoing or terminated by
> + * reading @kdamond. Reads and writes to @kdamond and @kdamond_stop from
> + * outside of the monitoring thread must be protected by @kdamond_lock.
> + *
> + * Note that the monitoring thread protects only @kdamond and @kdamond_stop via
> + * @kdamond_lock. Accesses to other fields must be protected by themselves.
> + *
> + * @primitive: Set of monitoring primitives for given use cases.
> + * @callback: Set of callbacks for monitoring events notifications.
> + *
> + * @target: Pointer to the user-defined monitoring target.
> + */
> +struct damon_ctx {
> + unsigned long sample_interval;
> + unsigned long aggr_interval;
> + unsigned long regions_update_interval;
> +
> +/* private */
> + struct timespec64 last_aggregation;
> + struct timespec64 last_regions_update;
> +
> +/* public */
> + struct task_struct *kdamond;
> + bool kdamond_stop;
> + struct mutex kdamond_lock;
> +
> + struct damon_primitive primitive;
> + struct damon_callback callback;
> +
> + void *target;
> +};
> +
> +#ifdef CONFIG_DAMON
> +
> +struct damon_ctx *damon_new_ctx(void);
> +void damon_destroy_ctx(struct damon_ctx *ctx);
> +int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> + unsigned long aggr_int, unsigned long regions_update_int);
> +
> +int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
> +int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
> +
> +#endif /* CONFIG_DAMON */
> +
> +#endif /* _DAMON_H */
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 390165ffbb0f..b97f2e8ab83f 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -859,4 +859,6 @@ config ARCH_HAS_HUGEPD
> config MAPPING_DIRTY_HELPERS
> bool
>
> +source "mm/damon/Kconfig"
> +
> endmenu
> diff --git a/mm/Makefile b/mm/Makefile
> index d73aed0fc99c..8022b8f04096 100644
> --- a/mm/Makefile
> +++ b/mm/Makefile
> @@ -120,3 +120,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o
> obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o
> obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
> obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
> +obj-$(CONFIG_DAMON) += damon/
> diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
> new file mode 100644
> index 000000000000..d00e99ac1a15
> --- /dev/null
> +++ b/mm/damon/Kconfig
> @@ -0,0 +1,15 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +menu "Data Access Monitoring"
> +
> +config DAMON
> + bool "DAMON: Data Access Monitoring Framework"
> + help
> + This builds a framework that allows kernel subsystems to monitor
> + access frequency of each memory region. The information can be useful
> + for performance-centric DRAM level memory management.
> +
> + See https://damonitor.github.io/doc/html/latest-damon/index.html for
> + more information.
> +
> +endmenu
> diff --git a/mm/damon/Makefile b/mm/damon/Makefile
> new file mode 100644
> index 000000000000..4fd2edb4becf
> --- /dev/null
> +++ b/mm/damon/Makefile
> @@ -0,0 +1,3 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +obj-$(CONFIG_DAMON) := core.o
> diff --git a/mm/damon/core.c b/mm/damon/core.c
> new file mode 100644
> index 000000000000..8963804efdf9
> --- /dev/null
> +++ b/mm/damon/core.c
> @@ -0,0 +1,316 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Data Access Monitor
> + *
> + * Author: SeongJae Park <[email protected]>
> + */
> +
> +#define pr_fmt(fmt) "damon: " fmt
> +
> +#include <linux/damon.h>
> +#include <linux/delay.h>
> +#include <linux/kthread.h>
> +#include <linux/slab.h>
> +
> +static DEFINE_MUTEX(damon_lock);
> +static int nr_running_ctxs;
> +
> +struct damon_ctx *damon_new_ctx(void)
> +{
> + struct damon_ctx *ctx;
> +
> + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> + if (!ctx)
> + return NULL;
> +
> + ctx->sample_interval = 5 * 1000;
> + ctx->aggr_interval = 100 * 1000;
> + ctx->regions_update_interval = 1000 * 1000;
> +
> + ktime_get_coarse_ts64(&ctx->last_aggregation);
> + ctx->last_regions_update = ctx->last_aggregation;
> +
> + mutex_init(&ctx->kdamond_lock);
> +
> + ctx->target = NULL;
> +
> + return ctx;
> +}
> +
> +void damon_destroy_ctx(struct damon_ctx *ctx)
> +{
> + kfree(ctx);
> +}
> +
> +/**
> + * damon_set_attrs() - Set attributes for the monitoring.
> + * @ctx: monitoring context
> + * @sample_int: time interval between samplings
> + * @aggr_int: time interval between aggregations
> + * @regions_update_int: time interval between target regions update
> + *
> + * This function should not be called while the kdamond is running.
> + * Every time interval is in micro-seconds.
> + *
> + * Return: 0 on success, negative error code otherwise.
> + */
> +int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> + unsigned long aggr_int, unsigned long regions_update_int)
> +{
> + ctx->sample_interval = sample_int;
> + ctx->aggr_interval = aggr_int;
> + ctx->regions_update_interval = regions_update_int;
> +
> + return 0;
> +}
> +
> +static bool damon_kdamond_running(struct damon_ctx *ctx)
> +{
> + bool running;
> +
> + mutex_lock(&ctx->kdamond_lock);
> + running = ctx->kdamond != NULL;
> + mutex_unlock(&ctx->kdamond_lock);
> +
> + return running;
> +}
> +
> +static int kdamond_fn(void *data);
> +
> +/*
> + * __damon_start() - Starts monitoring with given context.
> + * @ctx: monitoring context
> + *
> + * This function should be called while damon_lock is hold.
> + *
> + * Return: 0 on success, negative error code otherwise.
> + */
> +static int __damon_start(struct damon_ctx *ctx)
> +{
> + int err = -EBUSY;
> +
> + mutex_lock(&ctx->kdamond_lock);
> + if (!ctx->kdamond) {
> + err = 0;
> + ctx->kdamond_stop = false;
> + ctx->kdamond = kthread_create(kdamond_fn, ctx, "kdamond.%d",
> + nr_running_ctxs);
> + if (IS_ERR(ctx->kdamond))
> + err = PTR_ERR(ctx->kdamond);
> + else
> + wake_up_process(ctx->kdamond);
> + }
> + mutex_unlock(&ctx->kdamond_lock);
> +
> + return err;
> +}
> +
> +/**
> + * damon_start() - Starts the monitorings for a given group of contexts.
> + * @ctxs: an array of the pointers for contexts to start monitoring
> + * @nr_ctxs: size of @ctxs
> + *
> + * This function starts a group of monitoring threads for a group of monitoring
> + * contexts. One thread per each context is created and run in parallel. The
> + * caller should handle synchronization between the threads by itself. If a
> + * group of threads that created by other 'damon_start()' call is currently
> + * running, this function does nothing but returns -EBUSY.
> + *
> + * Return: 0 on success, negative error code otherwise.
> + */
> +int damon_start(struct damon_ctx **ctxs, int nr_ctxs)
> +{
> + int i;
> + int err = 0;
> +
> + mutex_lock(&damon_lock);
> + if (nr_running_ctxs) {
> + mutex_unlock(&damon_lock);
> + return -EBUSY;
> + }
> +
> + for (i = 0; i < nr_ctxs; i++) {
> + err = __damon_start(ctxs[i]);
> + if (err)
> + break;
> + nr_running_ctxs++;
> + }
> + mutex_unlock(&damon_lock);
> +
> + return err;
> +}
> +
> +/*
> + * __damon_stop() - Stops monitoring of given context.
> + * @ctx: monitoring context
> + *
> + * Return: 0 on success, negative error code otherwise.
> + */
> +static int __damon_stop(struct damon_ctx *ctx)
> +{
> + mutex_lock(&ctx->kdamond_lock);
> + if (ctx->kdamond) {
> + ctx->kdamond_stop = true;
> + mutex_unlock(&ctx->kdamond_lock);
> + while (damon_kdamond_running(ctx))
> + usleep_range(ctx->sample_interval,
> + ctx->sample_interval * 2);
> + return 0;
> + }
> + mutex_unlock(&ctx->kdamond_lock);
> +
> + return -EPERM;
> +}
> +
> +/**
> + * damon_stop() - Stops the monitorings for a given group of contexts.
> + * @ctxs: an array of the pointers for contexts to stop monitoring
> + * @nr_ctxs: size of @ctxs
> + *
> + * Return: 0 on success, negative error code otherwise.
> + */
> +int damon_stop(struct damon_ctx **ctxs, int nr_ctxs)
> +{
> + int i, err = 0;
> +
> + for (i = 0; i < nr_ctxs; i++) {
> + /* nr_running_ctxs is decremented in kdamond_fn */
> + err = __damon_stop(ctxs[i]);
> + if (err)
> + return err;
> + }
> +
> + return err;
> +}
> +
> +/*
> + * damon_check_reset_time_interval() - Check if a time interval is elapsed.
> + * @baseline: the time to check whether the interval has elapsed since
> + * @interval: the time interval (microseconds)
> + *
> + * See whether the given time interval has passed since the given baseline
> + * time. If so, it also updates the baseline to current time for next check.
> + *
> + * Return: true if the time interval has passed, or false otherwise.
> + */
> +static bool damon_check_reset_time_interval(struct timespec64 *baseline,
> + unsigned long interval)
> +{
> + struct timespec64 now;
> +
> + ktime_get_coarse_ts64(&now);
> + if ((timespec64_to_ns(&now) - timespec64_to_ns(baseline)) <
> + interval * 1000)
> + return false;
> + *baseline = now;
> + return true;
> +}
> +
> +/*
> + * Check whether it is time to flush the aggregated information
> + */
> +static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx)
> +{
> + return damon_check_reset_time_interval(&ctx->last_aggregation,
> + ctx->aggr_interval);
> +}
> +
> +/*
> + * Check whether it is time to check and apply the target monitoring regions
> + *
> + * Returns true if it is.
> + */
> +static bool kdamond_need_update_regions(struct damon_ctx *ctx)
> +{
> + return damon_check_reset_time_interval(&ctx->last_regions_update,
> + ctx->regions_update_interval);
> +}
> +
> +/*
> + * Check whether current monitoring should be stopped
> + *
> + * The monitoring is stopped when either the user requested to stop, or all
> + * monitoring targets are invalid.
> + *
> + * Returns true if need to stop current monitoring.
> + */
> +static bool kdamond_need_stop(struct damon_ctx *ctx)
> +{
> + bool stop;
> +
> + mutex_lock(&ctx->kdamond_lock);
> + stop = ctx->kdamond_stop;
> + mutex_unlock(&ctx->kdamond_lock);
> + if (stop)
> + return true;
> +
> + if (!ctx->primitive.target_valid)
> + return false;
> +
> + return !ctx->primitive.target_valid(ctx->target);
> +}
> +
> +static void set_kdamond_stop(struct damon_ctx *ctx)
> +{
> + mutex_lock(&ctx->kdamond_lock);
> + ctx->kdamond_stop = true;
> + mutex_unlock(&ctx->kdamond_lock);
> +}
> +
> +/*
> + * The monitoring daemon that runs as a kernel thread
> + */
> +static int kdamond_fn(void *data)
> +{
> + struct damon_ctx *ctx = (struct damon_ctx *)data;
> +
> + pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
> +
> + if (ctx->primitive.init_target_regions)
> + ctx->primitive.init_target_regions(ctx);
> + if (ctx->callback.before_start && ctx->callback.before_start(ctx))
> + set_kdamond_stop(ctx);
> +
> + while (!kdamond_need_stop(ctx)) {
> + if (ctx->primitive.prepare_access_checks)
> + ctx->primitive.prepare_access_checks(ctx);
> + if (ctx->callback.after_sampling &&
> + ctx->callback.after_sampling(ctx))
> + set_kdamond_stop(ctx);
Is 'break' needed here? Or do we want to complete this iteration (same
for the set_kdamond_stop() below)?
> +
> + usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
> +
> + if (ctx->primitive.check_accesses)
> + ctx->primitive.check_accesses(ctx);
> +
> + if (kdamond_aggregate_interval_passed(ctx)) {
> + if (ctx->callback.after_aggregation &&
> + ctx->callback.after_aggregation(ctx))
> + set_kdamond_stop(ctx);
> + if (ctx->primitive.reset_aggregated)
> + ctx->primitive.reset_aggregated(ctx);
> + }
> +
The following 'if' can be done in ctx->primitive.update() callback.
> + if (kdamond_need_update_regions(ctx)) {
> + if (ctx->primitive.update_target_regions)
> + ctx->primitive.update_target_regions(ctx);
> + }
> + }
> +
> + if (ctx->callback.before_terminate &&
> + ctx->callback.before_terminate(ctx))
> + set_kdamond_stop(ctx);
> + if (ctx->primitive.cleanup)
> + ctx->primitive.cleanup(ctx);
> +
> + pr_debug("kdamond (%d) finishes\n", ctx->kdamond->pid);
> + mutex_lock(&ctx->kdamond_lock);
> + ctx->kdamond = NULL;
> + mutex_unlock(&ctx->kdamond_lock);
> +
> + mutex_lock(&damon_lock);
> + nr_running_ctxs--;
> + mutex_unlock(&damon_lock);
> +
> + do_exit(0);
> +}
> --
> 2.17.1
>
Overall the patch looks good to me. Two concerns I have are if we
should damon_callback here or with the real user and the regions part
of primitive abstraction. For the first one, I don't have any strong
opinion but for the second one I do.
More specifically the question is if sampling and adaptive region
adjustment are general enough to be part of core monitoring context?
Can you give an example of a different primitive/use-case where these
would be beneficial.
On Tue, Dec 15, 2020 at 3:57 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> PG_idle and PG_young allow the two PTE Accessed bit users, Idle Page
> Tracking and the reclaim logic concurrently work while don't interfere
> each other. That is, when they need to clear the Accessed bit, they set
> PG_young to represent the previous state of the bit, respectively. And
> when they need to read the bit, if the bit is cleared, they further read
> the PG_young to know whether the other has cleared the bit meanwhile or
> not.
>
> We could add another page flag and extend the mechanism to use the flag
> if we need to add another concurrent PTE Accessed bit user subsystem.
> However, the space is limited. Meanwhile, if the new subsystem is
> mutually exclusive with IDLE_PAGE_TRACKING or interfering with it is not
> a real problem, it would be ok to simply reuse the PG_idle flag.
> However, it's impossible because the flags are dependent on
> IDLE_PAGE_TRACKING.
>
> To allow such reuse of the flags, this commit separates the PG_young and
> PG_idle flag logic from IDLE_PAGE_TRACKING and introduces new kernel
> config, 'PAGE_IDLE_FLAG'. Hence, a new subsystem would be able to reuse
> PG_idle without depending on IDLE_PAGE_TRACKING.
>
> In the next commit, DAMON's reference implementation of the virtual
> memory address space monitoring primitives will use it.
>
> Signed-off-by: SeongJae Park <[email protected]>
Reviewed-by: Shakeel Butt <[email protected]>
On Tue, Dec 15, 2020 at 3:58 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> This commit introduces a reference implementation of the address space
> specific low level primitives for the virtual address space, so that
> users of DAMON can easily monitor the data accesses on virtual address
> spaces of specific processes by simply configuring the implementation to
> be used by DAMON.
>
> The low level primitives for the fundamental access monitoring are
> defined in two parts:
>
> 1. Identification of the monitoring target address range for the address
> space.
> 2. Access check of specific address range in the target space.
>
> The reference implementation for the virtual address space does the
> works as below.
>
> PTE Accessed-bit Based Access Check
> -----------------------------------
>
> The implementation uses PTE Accessed-bit for basic access checks. That
> is, it clears the bit for next sampling target page and checks whether
'for the next'
> it set again after one sampling period. This could disturb the reclaim
'it is set'
> logic. DAMON uses ``PG_idle`` and ``PG_young`` page flags to solve the
> conflict, as Idle page tracking does.
>
> VMA-based Target Address Range Construction
> -------------------------------------------
>
> Only small parts in the super-huge virtual address space of the
> processes are mapped to physical memory and accessed. Thus, tracking
> the unmapped address regions is just wasteful. However, because DAMON
> can deal with some level of noise using the adaptive regions adjustment
> mechanism, tracking every mapping is not strictly required but could
> even incur a high overhead in some cases. That said, too huge unmapped
> areas inside the monitoring target should be removed to not take the
> time for the adaptive mechanism.
>
> For the reason, this implementation converts the complex mappings to
> three distinct regions that cover every mapped area of the address
> space. Also, the two gaps between the three regions are the two biggest
> unmapped areas in the given address space. The two biggest unmapped
> areas would be the gap between the heap and the uppermost mmap()-ed
> region, and the gap between the lowermost mmap()-ed region and the stack
> in most of the cases. Because these gaps are exceptionally huge in
> usual address spacees, excluding these will be sufficient to make a
*spaces
> reasonable trade-off. Below shows this in detail::
>
> <heap>
> <BIG UNMAPPED REGION 1>
> <uppermost mmap()-ed region>
> (small mmap()-ed regions and munmap()-ed regions)
> <lowermost mmap()-ed region>
> <BIG UNMAPPED REGION 2>
> <stack>
>
> Signed-off-by: SeongJae Park <[email protected]>
> Reviewed-by: Leonard Foerster <[email protected]>
> ---
> include/linux/damon.h | 13 +
> mm/damon/Kconfig | 9 +
> mm/damon/Makefile | 1 +
> mm/damon/vaddr.c | 579 ++++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 602 insertions(+)
> create mode 100644 mm/damon/vaddr.c
>
> diff --git a/include/linux/damon.h b/include/linux/damon.h
> index f446f8433599..39b4d6d3ddee 100644
> --- a/include/linux/damon.h
> +++ b/include/linux/damon.h
> @@ -274,4 +274,17 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
>
> #endif /* CONFIG_DAMON */
>
> +#ifdef CONFIG_DAMON_VADDR
> +
> +/* Monitoring primitives for virtual memory address spaces */
> +void damon_va_init_regions(struct damon_ctx *ctx);
> +void damon_va_update_regions(struct damon_ctx *ctx);
> +void damon_va_prepare_access_checks(struct damon_ctx *ctx);
> +unsigned int damon_va_check_accesses(struct damon_ctx *ctx);
> +bool damon_va_target_valid(void *t);
> +void damon_va_cleanup(struct damon_ctx *ctx);
> +void damon_va_set_primitives(struct damon_ctx *ctx);
Any reason for these to be in the header?
> +
[snip]
> +
> +static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
> + unsigned long *page_sz)
> +{
> + pte_t *pte = NULL;
> + pmd_t *pmd = NULL;
> + spinlock_t *ptl;
> + bool young = false;
> +
> + if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
> + return false;
> +
> + *page_sz = PAGE_SIZE;
> + if (pte) {
> + young = pte_young(*pte);
> + if (!young)
> + young = !page_is_idle(pte_page(*pte));
> + pte_unmap_unlock(pte, ptl);
> + return young;
> + }
> +
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> + young = pmd_young(*pmd);
> + if (!young)
> + young = !page_is_idle(pmd_page(*pmd));
> + spin_unlock(ptl);
> + *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
> +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> +
> + return young;
You need mmu_notifier_test_young() here. Hmm I remember mentioning
this in some previous version as well.
BTW have you tested this on a VM?
The patch looks good overall.
Thanks for the valuable comments, Shakeel!
On Wed, 23 Dec 2020 07:11:12 -0800 Shakeel Butt <[email protected]> wrote:
> First I would like you to prune your To/CC list.
I will remove people not directly related with this work and didn't comment to
this series yet.
>
> On Tue, Dec 15, 2020 at 3:56 AM SeongJae Park <[email protected]> wrote:
> >
> > From: SeongJae Park <[email protected]>
> >
> > DAMON is a data access monitoring framework for the Linux kernel. The
> > core mechanisms of DAMON make it
> >
> > - accurate (the monitoring output is useful enough for DRAM level
> > performance-centric memory management; It might be inappropriate for
> > CPU cache levels, though),
> > - light-weight (the monitoring overhead is normally low enough to be
> > applied online), and
> > - scalable (the upper-bound of the overhead is in constant range
> > regardless of the size of target workloads).
> >
> > Using this framework, hence, we can easily write efficient kernel space
> > data access monitoring applications. For example, the kernel's memory
> > management mechanisms can make advanced decisions using this.
> > Experimental data access aware optimization works that incurring high
> > access monitoring overhead could implemented again on top of this.
>
> *could again be implemented on*
Good eye! I will fix this in the next version.
>
> >
> > Due to its simple and flexible interface, providing user space interface
> > would be also easy. Then, user space users who have some special
> > workloads can write personalized applications for better understanding
> > and optimizations of their workloads and systems.
> >
> > ---
> >
> > Nevertheless, this commit is defining and implementing only basic access
> > check part without the overhead-accuracy handling core logic. The basic
> > access check is as below.
> >
> > The output of DAMON says what memory regions are how frequently accessed
> > for a given duration. The resolution of the access frequency is
> > controlled by setting ``sampling interval`` and ``aggregation
> > interval``. In detail, DAMON checks access to each page per ``sampling
> > interval`` and aggregates the results. In other words, counts the
> > number of the accesses to each region. After each ``aggregation
> > interval`` passes, DAMON calls callback functions that previously
> > registered by users so that users can read the aggregated results and
> > then clears the results. This can be described in below simple
> > pseudo-code::
> >
> > while monitoring_on:
> > for page in monitoring_target:
> > if accessed(page):
> > nr_accesses[page] += 1
> > if time() % aggregation_interval == 0:
> > for callback in user_registered_callbacks:
> > callback(monitoring_target, nr_accesses)
> > for page in monitoring_target:
> > nr_accesses[page] = 0
> > sleep(sampling interval)
> >
>
> The above is a good example and I was hoping to see the almost same actual code.
Oh, this doesn't explain about the 'update'. I will update this in the next
version.
>
> > The target regions constructed at the beginning of the monitoring and
> > updated after each ``regions_update_interval``, because the target
> > regions could be dynamically changed (e.g., mmap() or memory hotplug).
> > The monitoring overhead of this mechanism will arbitrarily increase as
> > the size of the target workload grows.
> >
> > The basic monitoring primitives for actual access check and dynamic
> > target regions construction aren't in the core part of DAMON. Instead,
> > it allows users to implement their own primitives that optimized for
>
> 'that are optimized for'
Good catch! I will fix this in the next version.
>
> > their use case and configure DAMON to use those. In other words, users
> > cannot use current version of DAMON without some additional works.
> >
> > Following commits will implement the core mechanisms for the
> > overhead-accuracy control and default primitives implementations.
> >
> > Signed-off-by: SeongJae Park <[email protected]>
> > Reviewed-by: Leonard Foerster <[email protected]>
> > ---
> > include/linux/damon.h | 167 ++++++++++++++++++++++
> > mm/Kconfig | 2 +
> > mm/Makefile | 1 +
> > mm/damon/Kconfig | 15 ++
> > mm/damon/Makefile | 3 +
> > mm/damon/core.c | 316 ++++++++++++++++++++++++++++++++++++++++++
> > 6 files changed, 504 insertions(+)
> > create mode 100644 include/linux/damon.h
> > create mode 100644 mm/damon/Kconfig
> > create mode 100644 mm/damon/Makefile
> > create mode 100644 mm/damon/core.c
> >
> > diff --git a/include/linux/damon.h b/include/linux/damon.h
> > new file mode 100644
> > index 000000000000..387fa4399fc8
> > --- /dev/null
> > +++ b/include/linux/damon.h
> > @@ -0,0 +1,167 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * DAMON api
> > + *
> > + * Author: SeongJae Park <[email protected]>
> > + */
> > +
> > +#ifndef _DAMON_H_
> > +#define _DAMON_H_
> > +
> > +#include <linux/mutex.h>
> > +#include <linux/time64.h>
> > +#include <linux/types.h>
> > +
> > +struct damon_ctx;
> > +
> > +/**
> > + * struct damon_primitive Monitoring primitives for given use cases.
> > + *
> > + * @init_target_regions: Constructs initial monitoring target regions.
> > + * @update_target_regions: Updates monitoring target regions.
>
> The 'region' here still does not feel right. I prefer to rename the
> above two to just init() and update().
Agreed. I will rename so in the next version.
>
> > + * @prepare_access_checks: Prepares next access check of target regions.
> > + * @check_accesses: Checks the access of target regions.
> > + * @reset_aggregated: Resets aggregated accesses monitoring results.
> > + * @target_valid: Determine if the target is valid.
> > + * @cleanup: Cleans up the context.
> > + *
> > + * DAMON can be extended for various address spaces and usages. For this,
> > + * users should register the low level primitives for their target address
> > + * space and usecase via the &damon_ctx.primitive. Then, the monitoring thread
> > + * calls @init_target_regions and @prepare_access_checks before starting the
> > + * monitoring, @update_target_regions after each
> > + * &damon_ctx.regions_update_interval, and @check_accesses, @target_valid and
> > + * @prepare_access_checks after each &damon_ctx.sample_interval. Finally,
> > + * @reset_aggregated is called after each &damon_ctx.aggr_interval.
> > + *
> > + * @init_target_regions should construct proper monitoring target regions and
> > + * link those to the DAMON context struct. The regions should be defined by
> > + * user and saved in @damon_ctx.target.
> > + * @update_target_regions should update the monitoring target regions for
> > + * current status.
> > + * @prepare_access_checks should manipulate the monitoring regions to be
> > + * prepared for the next access check.
> > + * @check_accesses should check the accesses to each region that made after the
> > + * last preparation and update the number of observed accesses of each region.
> > + * @reset_aggregated should reset the access monitoring results that aggregated
> > + * by @check_accesses.
> > + * @target_valid should check whether the target is still valid for the
> > + * monitoring.
> > + * @cleanup is called from @kdamond just before its termination. After this
> > + * call, only @kdamond_lock and @kdamond will be touched.
> > + */
> > +struct damon_primitive {
> > + void (*init_target_regions)(struct damon_ctx *context);
> > + void (*update_target_regions)(struct damon_ctx *context);
> > + void (*prepare_access_checks)(struct damon_ctx *context);
> > + void (*check_accesses)(struct damon_ctx *context);
> > + void (*reset_aggregated)(struct damon_ctx *context);
> > + bool (*target_valid)(void *target);
> > + void (*cleanup)(struct damon_ctx *context);
> > +};
> > +
> > +/*
> > + * struct damon_callback Monitoring events notification callbacks.
> > + *
> > + * @before_start: Called before starting the monitoring.
> > + * @after_sampling: Called after each sampling.
> > + * @after_aggregation: Called after each aggregation.
> > + * @before_terminate: Called before terminating the monitoring.
> > + * @private: User private data.
> > + *
> > + * The monitoring thread (&damon_ctx->kdamond) calls @before_start and
> > + * @before_terminate just before starting and finishing the monitoring,
> > + * respectively. Therefore, those are good places for installing and cleaning
> > + * @private.
> > + *
> > + * The monitoring thread calls @after_sampling and @after_aggregation for each
> > + * of the sampling intervals and aggregation intervals, respectively.
> > + * Therefore, users can safely access the monitoring results without additional
> > + * protection. For the reason, users are recommended to use these callback for
> > + * the accesses to the results.
> > + *
> > + * If any callback returns non-zero, monitoring stops.
>
> I am not sure if this is the right patch to add this struct. Either we
> need to add text why this is needed or add this patch when the real
> user i.e. debugfs interface is added.
I think this is better to be here to let people know how DAMON API users can
access to the monitoring result.
>
> > + */
> > +struct damon_callback {
> > + void *private;
> > +
> > + int (*before_start)(struct damon_ctx *context);
> > + int (*after_sampling)(struct damon_ctx *context);
> > + int (*after_aggregation)(struct damon_ctx *context);
> > + int (*before_terminate)(struct damon_ctx *context);
> > +};
> > +
> > +/**
> > + * struct damon_ctx - Represents a context for each monitoring. This is the
> > + * main interface that allows users to set the attributes and get the results
> > + * of the monitoring.
> > + *
> > + * @sample_interval: The time between access samplings.
> > + * @aggr_interval: The time between monitor results aggregations.
> > + * @regions_update_interval: The time between monitor regions updates.
>
> regions_update_internal should be part of the primitive abstraction
> and the update() callback internally can check this field.
I think the field should be renamed to be independent from the 'region'
concept.
However, if we agree that the 'update()' is general concept (I assume
so because you didn't object to adding 'update()' callback), I believe this
should be here, because the interval is the part of the monitoring request.
Also, I unsure how the monitoring thread could know when to call 'update()'
callback if this field is not here.
[...]
> > diff --git a/mm/Kconfig b/mm/Kconfig
> > index 390165ffbb0f..b97f2e8ab83f 100644
> > --- a/mm/Kconfig
> > +++ b/mm/Kconfig
> > @@ -859,4 +859,6 @@ config ARCH_HAS_HUGEPD
> > config MAPPING_DIRTY_HELPERS
> > bool
> >
> > +source "mm/damon/Kconfig"
> > +
> > endmenu
> > diff --git a/mm/Makefile b/mm/Makefile
> > index d73aed0fc99c..8022b8f04096 100644
> > --- a/mm/Makefile
> > +++ b/mm/Makefile
> > @@ -120,3 +120,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o
> > obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o
> > obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
> > obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
> > +obj-$(CONFIG_DAMON) += damon/
> > diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
> > new file mode 100644
> > index 000000000000..d00e99ac1a15
> > --- /dev/null
> > +++ b/mm/damon/Kconfig
> > @@ -0,0 +1,15 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +
> > +menu "Data Access Monitoring"
> > +
> > +config DAMON
> > + bool "DAMON: Data Access Monitoring Framework"
> > + help
> > + This builds a framework that allows kernel subsystems to monitor
> > + access frequency of each memory region. The information can be useful
> > + for performance-centric DRAM level memory management.
> > +
> > + See https://damonitor.github.io/doc/html/latest-damon/index.html for
> > + more information.
> > +
> > +endmenu
> > diff --git a/mm/damon/Makefile b/mm/damon/Makefile
> > new file mode 100644
> > index 000000000000..4fd2edb4becf
> > --- /dev/null
> > +++ b/mm/damon/Makefile
> > @@ -0,0 +1,3 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +obj-$(CONFIG_DAMON) := core.o
> > diff --git a/mm/damon/core.c b/mm/damon/core.c
> > new file mode 100644
> > index 000000000000..8963804efdf9
> > --- /dev/null
> > +++ b/mm/damon/core.c
[...]
> > +/*
> > + * The monitoring daemon that runs as a kernel thread
> > + */
> > +static int kdamond_fn(void *data)
> > +{
> > + struct damon_ctx *ctx = (struct damon_ctx *)data;
> > +
> > + pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
> > +
> > + if (ctx->primitive.init_target_regions)
> > + ctx->primitive.init_target_regions(ctx);
> > + if (ctx->callback.before_start && ctx->callback.before_start(ctx))
> > + set_kdamond_stop(ctx);
> > +
> > + while (!kdamond_need_stop(ctx)) {
> > + if (ctx->primitive.prepare_access_checks)
> > + ctx->primitive.prepare_access_checks(ctx);
> > + if (ctx->callback.after_sampling &&
> > + ctx->callback.after_sampling(ctx))
> > + set_kdamond_stop(ctx);
>
> Is 'break' needed here? Or do we want to complete this iteration (same
> for the set_kdamond_stop() below)?
My intention was to complete this iteration. I think it could give slightly
simpler view to primitives and callbacks implementing people, because they can
assume callbacks in each iteration will be called.
>
> > +
> > + usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
> > +
> > + if (ctx->primitive.check_accesses)
> > + ctx->primitive.check_accesses(ctx);
> > +
> > + if (kdamond_aggregate_interval_passed(ctx)) {
> > + if (ctx->callback.after_aggregation &&
> > + ctx->callback.after_aggregation(ctx))
> > + set_kdamond_stop(ctx);
> > + if (ctx->primitive.reset_aggregated)
> > + ctx->primitive.reset_aggregated(ctx);
> > + }
> > +
>
> The following 'if' can be done in ctx->primitive.update() callback.
I think this could be common to multiple primitives, so I'd like to let it be
here to minimize code duplication.
>
> > + if (kdamond_need_update_regions(ctx)) {
> > + if (ctx->primitive.update_target_regions)
> > + ctx->primitive.update_target_regions(ctx);
> > + }
> > + }
> > +
> > + if (ctx->callback.before_terminate &&
> > + ctx->callback.before_terminate(ctx))
> > + set_kdamond_stop(ctx);
> > + if (ctx->primitive.cleanup)
> > + ctx->primitive.cleanup(ctx);
> > +
> > + pr_debug("kdamond (%d) finishes\n", ctx->kdamond->pid);
> > + mutex_lock(&ctx->kdamond_lock);
> > + ctx->kdamond = NULL;
> > + mutex_unlock(&ctx->kdamond_lock);
> > +
> > + mutex_lock(&damon_lock);
> > + nr_running_ctxs--;
> > + mutex_unlock(&damon_lock);
> > +
> > + do_exit(0);
> > +}
> > --
> > 2.17.1
> >
>
> Overall the patch looks good to me. Two concerns I have are if we
> should damon_callback here or with the real user and the regions part
> of primitive abstraction. For the first one, I don't have any strong
> opinion but for the second one I do.
I'd like to keep 'damon_callback' part here, to let API users know how the
monitoring result will be available to them.
For the 'regions' part, I will rename relevant things as below in the next
version, to reduce any confusion.
init_target_regions() -> init()
update_target_regions() -> update()
regions_update_interval -> update_interval
last_regions_update -> last_update
>
> More specifically the question is if sampling and adaptive region
> adjustment are general enough to be part of core monitoring context?
> Can you give an example of a different primitive/use-case where these
> would be beneficial.
I think all adress spaces having some spatial locality and monitoring requests
that need to have upper-bound overhead and best-effort accuracy could get
benefit from it. The primitives targetting 'virtual address spaces' and the
'physical address space' clearly showed the benefit.
On Wed, 23 Dec 2020 07:31:45 -0800 Shakeel Butt <[email protected]> wrote:
> On Tue, Dec 15, 2020 at 3:58 AM SeongJae Park <[email protected]> wrote:
> >
> > From: SeongJae Park <[email protected]>
> >
> > This commit introduces a reference implementation of the address space
> > specific low level primitives for the virtual address space, so that
> > users of DAMON can easily monitor the data accesses on virtual address
> > spaces of specific processes by simply configuring the implementation to
> > be used by DAMON.
> >
> > The low level primitives for the fundamental access monitoring are
> > defined in two parts:
> >
> > 1. Identification of the monitoring target address range for the address
> > space.
> > 2. Access check of specific address range in the target space.
> >
> > The reference implementation for the virtual address space does the
> > works as below.
> >
> > PTE Accessed-bit Based Access Check
> > -----------------------------------
> >
> > The implementation uses PTE Accessed-bit for basic access checks. That
> > is, it clears the bit for next sampling target page and checks whether
>
> 'for the next'
>
> > it set again after one sampling period. This could disturb the reclaim
>
> 'it is set'
Good catch! Will fix in the next version.
>
> > logic. DAMON uses ``PG_idle`` and ``PG_young`` page flags to solve the
> > conflict, as Idle page tracking does.
> >
> > VMA-based Target Address Range Construction
> > -------------------------------------------
> >
> > Only small parts in the super-huge virtual address space of the
> > processes are mapped to physical memory and accessed. Thus, tracking
> > the unmapped address regions is just wasteful. However, because DAMON
> > can deal with some level of noise using the adaptive regions adjustment
> > mechanism, tracking every mapping is not strictly required but could
> > even incur a high overhead in some cases. That said, too huge unmapped
> > areas inside the monitoring target should be removed to not take the
> > time for the adaptive mechanism.
> >
> > For the reason, this implementation converts the complex mappings to
> > three distinct regions that cover every mapped area of the address
> > space. Also, the two gaps between the three regions are the two biggest
> > unmapped areas in the given address space. The two biggest unmapped
> > areas would be the gap between the heap and the uppermost mmap()-ed
> > region, and the gap between the lowermost mmap()-ed region and the stack
> > in most of the cases. Because these gaps are exceptionally huge in
> > usual address spacees, excluding these will be sufficient to make a
>
> *spaces
Good eye! Will fix in the next version.
>
> > reasonable trade-off. Below shows this in detail::
> >
> > <heap>
> > <BIG UNMAPPED REGION 1>
> > <uppermost mmap()-ed region>
> > (small mmap()-ed regions and munmap()-ed regions)
> > <lowermost mmap()-ed region>
> > <BIG UNMAPPED REGION 2>
> > <stack>
> >
> > Signed-off-by: SeongJae Park <[email protected]>
> > Reviewed-by: Leonard Foerster <[email protected]>
> > ---
> > include/linux/damon.h | 13 +
> > mm/damon/Kconfig | 9 +
> > mm/damon/Makefile | 1 +
> > mm/damon/vaddr.c | 579 ++++++++++++++++++++++++++++++++++++++++++
> > 4 files changed, 602 insertions(+)
> > create mode 100644 mm/damon/vaddr.c
> >
> > diff --git a/include/linux/damon.h b/include/linux/damon.h
> > index f446f8433599..39b4d6d3ddee 100644
> > --- a/include/linux/damon.h
> > +++ b/include/linux/damon.h
> > @@ -274,4 +274,17 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
> >
> > #endif /* CONFIG_DAMON */
> >
> > +#ifdef CONFIG_DAMON_VADDR
> > +
> > +/* Monitoring primitives for virtual memory address spaces */
> > +void damon_va_init_regions(struct damon_ctx *ctx);
> > +void damon_va_update_regions(struct damon_ctx *ctx);
> > +void damon_va_prepare_access_checks(struct damon_ctx *ctx);
> > +unsigned int damon_va_check_accesses(struct damon_ctx *ctx);
> > +bool damon_va_target_valid(void *t);
> > +void damon_va_cleanup(struct damon_ctx *ctx);
> > +void damon_va_set_primitives(struct damon_ctx *ctx);
>
> Any reason for these to be in the header?
To let DAMON API users (in kernel space) and other primitives developers to use
those.
>
> > +
> [snip]
> > +
> > +static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
> > + unsigned long *page_sz)
> > +{
> > + pte_t *pte = NULL;
> > + pmd_t *pmd = NULL;
> > + spinlock_t *ptl;
> > + bool young = false;
> > +
> > + if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
> > + return false;
> > +
> > + *page_sz = PAGE_SIZE;
> > + if (pte) {
> > + young = pte_young(*pte);
> > + if (!young)
> > + young = !page_is_idle(pte_page(*pte));
> > + pte_unmap_unlock(pte, ptl);
> > + return young;
> > + }
> > +
> > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> > + young = pmd_young(*pmd);
> > + if (!young)
> > + young = !page_is_idle(pmd_page(*pmd));
> > + spin_unlock(ptl);
> > + *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
> > +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> > +
> > + return young;
>
> You need mmu_notifier_test_young() here. Hmm I remember mentioning
> this in some previous version as well.
Your question and my answer was as below:
> Don't you need mmu_notifier_clear_young() here?
I think we don't need it here because we only read the Accessed bit and PG_Idle
if Accessed bit was not set.
I should notice that you mean 'test_young()' but didn't, sorry. I will add it
in the next version.
>
> BTW have you tested this on a VM?
Yes. Indeed, I'm testing this on a QEMU/KVM environment. You can get more
detail at: https://damonitor.github.io/doc/html/latest/vm/damon/eval.html#setup
>
> The patch looks good overall.
Thanks. I will fix above things in the next version.
Thanks,
SeongJae Park
On Tue, 15 Dec 2020 12:54:44 +0100 SeongJae Park <[email protected]> wrote:
> From: SeongJae Park <[email protected]>
>
> 'damon-dbgfs' provides simple user space interface for DAMON, but using
> the interface for complex usages could require annoying repetitive
> works. Writing a user space data access monitoring applications on top
> of the debugfs interface and using the application would be better for
> such complex cases.
>
> This commit introduces a reference implementation of such user space
> application built on top of the debugfs interface, namely 'DAMon
> Operator' (DAMO). It contains a shallow wrapper python script of the
> debugfs interface and various visualization of the monitoring results
> convenient user interface.
>
> Note that it is initially aimed to be used for minimal reference of the
> 'damon-dbgfs' interface and for debugging of the DAMON itself.
>
> Signed-off-by: SeongJae Park <[email protected]>
> ---
> tools/damon/.gitignore | 1 +
> tools/damon/_damon.py | 130 ++++++++++++++
> tools/damon/_dist.py | 35 ++++
> tools/damon/_recfile.py | 23 +++
> tools/damon/bin2txt.py | 67 +++++++
> tools/damon/damo | 37 ++++
> tools/damon/heats.py | 362 ++++++++++++++++++++++++++++++++++++++
> tools/damon/nr_regions.py | 91 ++++++++++
> tools/damon/record.py | 135 ++++++++++++++
> tools/damon/report.py | 45 +++++
> tools/damon/wss.py | 100 +++++++++++
> 11 files changed, 1026 insertions(+)
I will drop this patch from the next version of this patchset, because this is
not the essential part but could only make the code review time consuming. I
will post another DAMON RFC patchset for convenient user space usages including
this and the perf integration.
On Wed, Dec 23, 2020 at 8:34 AM SeongJae Park <[email protected]> wrote:
[snip]
> > Overall the patch looks good to me. Two concerns I have are if we
> > should damon_callback here or with the real user and the regions part
> > of primitive abstraction. For the first one, I don't have any strong
> > opinion but for the second one I do.
>
> I'd like to keep 'damon_callback' part here, to let API users know how the
> monitoring result will be available to them.
>
> For the 'regions' part, I will rename relevant things as below in the next
> version, to reduce any confusion.
>
> init_target_regions() -> init()
> update_target_regions() -> update()
> regions_update_interval -> update_interval
> last_regions_update -> last_update
>
> >
> > More specifically the question is if sampling and adaptive region
> > adjustment are general enough to be part of core monitoring context?
> > Can you give an example of a different primitive/use-case where these
> > would be beneficial.
>
> I think all adress spaces having some spatial locality and monitoring requests
> that need to have upper-bound overhead and best-effort accuracy could get
> benefit from it. The primitives targetting 'virtual address spaces' and the
> 'physical address space' clearly showed the benefit.
I am still not much convinced on the 'physical address space' use-case
or the way you are presenting it. Anyways I think we start with what
you have and if in future there is a use-case where regions adjustment
does not make sense, we can change it then.
On Wed, Dec 23, 2020 at 8:47 AM SeongJae Park <[email protected]> wrote:
>
[snip]
> > [snip]
> > > +
> > > +static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
> > > + unsigned long *page_sz)
> > > +{
> > > + pte_t *pte = NULL;
> > > + pmd_t *pmd = NULL;
> > > + spinlock_t *ptl;
> > > + bool young = false;
> > > +
> > > + if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
> > > + return false;
> > > +
> > > + *page_sz = PAGE_SIZE;
> > > + if (pte) {
> > > + young = pte_young(*pte);
> > > + if (!young)
> > > + young = !page_is_idle(pte_page(*pte));
> > > + pte_unmap_unlock(pte, ptl);
> > > + return young;
> > > + }
> > > +
> > > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> > > + young = pmd_young(*pmd);
> > > + if (!young)
> > > + young = !page_is_idle(pmd_page(*pmd));
> > > + spin_unlock(ptl);
> > > + *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
> > > +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> > > +
> > > + return young;
> >
> > You need mmu_notifier_test_young() here. Hmm I remember mentioning
> > this in some previous version as well.
>
> Your question and my answer was as below:
>
> > Don't you need mmu_notifier_clear_young() here?
>
> I think we don't need it here because we only read the Accessed bit and PG_Idle
> if Accessed bit was not set.
>
> I should notice that you mean 'test_young()' but didn't, sorry. I will add it
> in the next version.
>
I should have said mmu_notifier_test_young() instead of
mmu_notifier_clear_young().
> >
> > BTW have you tested this on a VM?
>
> Yes. Indeed, I'm testing this on a QEMU/KVM environment. You can get more
> detail at: https://damonitor.github.io/doc/html/latest/vm/damon/eval.html#setup
>
Hmm without mmu_notifier_test_young() you should be missing the kvm
mmu access updates. Can you please recheck if your eval is correctly
seeing the memory accesses from the VM?
On Wed, Dec 23, 2020 at 10:38 AM SeongJae Park <[email protected]> wrote:
>
[snip]
>
> I will drop this patch from the next version of this patchset, because this is
> not the essential part but could only make the code review time consuming. I
> will post another DAMON RFC patchset for convenient user space usages including
> this and the perf integration.
You can just add the link to your github repo with commit/tag
containing the usage examples in the cover letter.
On Wed, 23 Dec 2020 14:49:57 -0800 Shakeel Butt <[email protected]> wrote:
> On Wed, Dec 23, 2020 at 8:34 AM SeongJae Park <[email protected]> wrote:
> [snip]
> > > Overall the patch looks good to me. Two concerns I have are if we
> > > should damon_callback here or with the real user and the regions part
> > > of primitive abstraction. For the first one, I don't have any strong
> > > opinion but for the second one I do.
> >
> > I'd like to keep 'damon_callback' part here, to let API users know how the
> > monitoring result will be available to them.
> >
> > For the 'regions' part, I will rename relevant things as below in the next
> > version, to reduce any confusion.
> >
> > init_target_regions() -> init()
> > update_target_regions() -> update()
> > regions_update_interval -> update_interval
> > last_regions_update -> last_update
> >
> > >
> > > More specifically the question is if sampling and adaptive region
> > > adjustment are general enough to be part of core monitoring context?
> > > Can you give an example of a different primitive/use-case where these
> > > would be beneficial.
> >
> > I think all adress spaces having some spatial locality and monitoring requests
> > that need to have upper-bound overhead and best-effort accuracy could get
> > benefit from it. The primitives targetting 'virtual address spaces' and the
> > 'physical address space' clearly showed the benefit.
>
> I am still not much convinced on the 'physical address space' use-case
> or the way you are presenting it.
I understand the concern. I also once thought the mechanism might not work
well for the physical address space because we cannot expect much spatial
locality in the space. However, it turned out that there is some (temporal)
spatial locality that enough to make DAMON work reasonably well. The word,
'reasonably well' might be controversial. With the mechanism, DAMON provides
only 'best-effort' accuracy, rather than 100% accuracy. Our goal is to make
the information accurate enough only for DRAM-centric optimizations. I'd like
to also note that there are knobs that you can use to make minimum quality
higher (nr_min_regions) while setting the upperbound of the monitoring overhead
(nr_max_regions). What I can say for now is that we ran DAMON for physical
address space of our production systems (shared detail in the 'Real-workd User
Story' section of coverletter[1]) and the result was reasonable enough to
convince the owner of the systems.
[1] https://lore.kernel.org/linux-mm/[email protected]/
> Anyways I think we start with what you have and if in future there is a
> use-case where regions adjustment does not make sense, we can change it then.
100% agreed, and thank you for understanding my argument.
Thanks,
SeongJae Park
On Wed, 23 Dec 2020 14:54:02 -0800 Shakeel Butt <[email protected]> wrote:
> On Wed, Dec 23, 2020 at 8:47 AM SeongJae Park <[email protected]> wrote:
> >
> [snip]
> > > [snip]
> > > > +
> > > > +static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
> > > > + unsigned long *page_sz)
> > > > +{
> > > > + pte_t *pte = NULL;
> > > > + pmd_t *pmd = NULL;
> > > > + spinlock_t *ptl;
> > > > + bool young = false;
> > > > +
> > > > + if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
> > > > + return false;
> > > > +
> > > > + *page_sz = PAGE_SIZE;
> > > > + if (pte) {
> > > > + young = pte_young(*pte);
> > > > + if (!young)
> > > > + young = !page_is_idle(pte_page(*pte));
> > > > + pte_unmap_unlock(pte, ptl);
> > > > + return young;
> > > > + }
> > > > +
> > > > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> > > > + young = pmd_young(*pmd);
> > > > + if (!young)
> > > > + young = !page_is_idle(pmd_page(*pmd));
> > > > + spin_unlock(ptl);
> > > > + *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
> > > > +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> > > > +
> > > > + return young;
> > >
> > > You need mmu_notifier_test_young() here. Hmm I remember mentioning
> > > this in some previous version as well.
> >
> > Your question and my answer was as below:
> >
> > > Don't you need mmu_notifier_clear_young() here?
> >
> > I think we don't need it here because we only read the Accessed bit and PG_Idle
> > if Accessed bit was not set.
> >
> > I should notice that you mean 'test_young()' but didn't, sorry. I will add it
> > in the next version.
> >
>
> I should have said mmu_notifier_test_young() instead of
> mmu_notifier_clear_young().
>
> > >
> > > BTW have you tested this on a VM?
> >
> > Yes. Indeed, I'm testing this on a QEMU/KVM environment. You can get more
> > detail at: https://damonitor.github.io/doc/html/latest/vm/damon/eval.html#setup
> >
>
> Hmm without mmu_notifier_test_young() you should be missing the kvm
> mmu access updates. Can you please recheck if your eval is correctly
> seeing the memory accesses from the VM?
Seems I didn't clearly answered, sorry. My test setup installs the
DAMON-enabled kernel in a guest VM and run it for workloads in the guest,
rather than running DAMON in host to monitor accesses of VMs. The MMU notifier
is for latter case, AFAIU, so my test setup didn't see the problem.
If I'm missing something, please let me know.
Thanks,
SeongJae Park
On Wed, 23 Dec 2020 14:56:10 -0800 Shakeel Butt <[email protected]> wrote:
> On Wed, Dec 23, 2020 at 10:38 AM SeongJae Park <[email protected]> wrote:
> >
> [snip]
> >
> > I will drop this patch from the next version of this patchset, because this is
> > not the essential part but could only make the code review time consuming. I
> > will post another DAMON RFC patchset for convenient user space usages including
> > this and the perf integration.
>
> You can just add the link to your github repo with commit/tag
> containing the usage examples in the cover letter.
Thank you for the nice suggestion! I will keep this patch on my development
tree at Github and provide a link to it.
Thanks,
SeongJae Park
On Tue, 15 Dec 2020 12:54:36 +0100 SeongJae Park <[email protected]> wrote:
> From: SeongJae Park <[email protected]>
>
> Even somehow the initial monitoring target regions are well constructed
> to fulfill the assumption (pages in same region have similar access
> frequencies), the data access pattern can be dynamically changed. This
> will result in low monitoring quality. To keep the assumption as much
> as possible, DAMON adaptively merges and splits each region based on
> their access frequency.
>
> For each ``aggregation interval``, it compares the access frequencies of
> adjacent regions and merges those if the frequency difference is small.
> Then, after it reports and clears the aggregated access frequency of
> each region, it splits each region into two or three regions if the
> total number of regions will not exceed the user-specified maximum
> number of regions after the split.
>
> In this way, DAMON provides its best-effort quality and minimal overhead
> while keeping the upper-bound overhead that users set.
>
> Signed-off-by: SeongJae Park <[email protected]>
> Reviewed-by: Leonard Foerster <[email protected]>
> ---
> include/linux/damon.h | 41 +++++---
> mm/damon/core.c | 220 ++++++++++++++++++++++++++++++++++++++++--
> 2 files changed, 240 insertions(+), 21 deletions(-)
>
[...]
> diff --git a/mm/damon/core.c b/mm/damon/core.c
> index 167487e75737..0f9beb60d9dd 100644
> --- a/mm/damon/core.c
> +++ b/mm/damon/core.c
[...]
> +
> +/*
> + * Split a region in two
> + *
> + * r the region to be split
> + * sz_r size of the first sub-region that will be made
> + */
> +static void damon_split_region_at(struct damon_ctx *ctx,
> + struct damon_region *r, unsigned long sz_r)
> +{
> + struct damon_region *new;
> +
> + new = damon_new_region(r->ar.start + sz_r, r->ar.end);
Coverity Static Analysis Security Testing (SAST) by Synopsys, Inc. found that
'damon_new_region()' could return NULL in case of memory allocation failure,
but NULL check for 'new' is missed here. I will add the check in the next
version.
> + r->ar.end = new->ar.start;
> +
> + damon_insert_region(new, r, damon_next_region(r));
> +}
[...]
Thanks,
SeongJae Park
On Wed, Jan 27, 2021 at 8:57 AM SeongJae Park <[email protected]> wrote:
>
> On Thu, 24 Dec 2020 08:11:11 +0100 SeongJae Park <[email protected]> wrote:
>
> > On Wed, 23 Dec 2020 14:54:02 -0800 Shakeel Butt <[email protected]> wrote:
> >
> > > On Wed, Dec 23, 2020 at 8:47 AM SeongJae Park <[email protected]> wrote:
> > > >
> > > [snip]
> > > > > [snip]
> > > > > > +
> > > > > > +static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
> > > > > > + unsigned long *page_sz)
> > > > > > +{
> > > > > > + pte_t *pte = NULL;
> > > > > > + pmd_t *pmd = NULL;
> > > > > > + spinlock_t *ptl;
> > > > > > + bool young = false;
> > > > > > +
> > > > > > + if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
> > > > > > + return false;
> > > > > > +
> > > > > > + *page_sz = PAGE_SIZE;
> > > > > > + if (pte) {
> > > > > > + young = pte_young(*pte);
> > > > > > + if (!young)
> > > > > > + young = !page_is_idle(pte_page(*pte));
> > > > > > + pte_unmap_unlock(pte, ptl);
> > > > > > + return young;
> > > > > > + }
> > > > > > +
> > > > > > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> > > > > > + young = pmd_young(*pmd);
> > > > > > + if (!young)
> > > > > > + young = !page_is_idle(pmd_page(*pmd));
> > > > > > + spin_unlock(ptl);
> > > > > > + *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
> > > > > > +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> > > > > > +
> > > > > > + return young;
> > > > >
> > > > > You need mmu_notifier_test_young() here. Hmm I remember mentioning
> > > > > this in some previous version as well.
> > > >
> > > > Your question and my answer was as below:
> > > >
> > > > > Don't you need mmu_notifier_clear_young() here?
> > > >
> > > > I think we don't need it here because we only read the Accessed bit and PG_Idle
> > > > if Accessed bit was not set.
> > > >
> > > > I should notice that you mean 'test_young()' but didn't, sorry. I will add it
> > > > in the next version.
> > > >
> > >
> > > I should have said mmu_notifier_test_young() instead of
> > > mmu_notifier_clear_young().
> > >
> > > > >
> > > > > BTW have you tested this on a VM?
> > > >
> > > > Yes. Indeed, I'm testing this on a QEMU/KVM environment. You can get more
> > > > detail at: https://damonitor.github.io/doc/html/latest/vm/damon/eval.html#setup
> > > >
> > >
> > > Hmm without mmu_notifier_test_young() you should be missing the kvm
> > > mmu access updates. Can you please recheck if your eval is correctly
> > > seeing the memory accesses from the VM?
> >
> > Seems I didn't clearly answered, sorry. My test setup installs the
> > DAMON-enabled kernel in a guest VM and run it for workloads in the guest,
> > rather than running DAMON in host to monitor accesses of VMs. The MMU notifier
> > is for latter case, AFAIU, so my test setup didn't see the problem.
>
> Just FYI. I confirmed the mmu_notifier_test_young() added version works for
> the use case. I tested it by running a program accessing 200MB memory in a
> QEMU/KVM guest having 120GB memory and monitoring the qemu process' virtual
> address space from the host using DAMON. The 200MB memory region was clearly
> identifiable.
>
Thanks for confirming. I am still going over the whole series and will
send out the emails in a couple of days.
Shakeel
On Thu, 24 Dec 2020 08:11:11 +0100 SeongJae Park <[email protected]> wrote:
> On Wed, 23 Dec 2020 14:54:02 -0800 Shakeel Butt <[email protected]> wrote:
>
> > On Wed, Dec 23, 2020 at 8:47 AM SeongJae Park <[email protected]> wrote:
> > >
> > [snip]
> > > > [snip]
> > > > > +
> > > > > +static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
> > > > > + unsigned long *page_sz)
> > > > > +{
> > > > > + pte_t *pte = NULL;
> > > > > + pmd_t *pmd = NULL;
> > > > > + spinlock_t *ptl;
> > > > > + bool young = false;
> > > > > +
> > > > > + if (follow_pte_pmd(mm, addr, NULL, &pte, &pmd, &ptl))
> > > > > + return false;
> > > > > +
> > > > > + *page_sz = PAGE_SIZE;
> > > > > + if (pte) {
> > > > > + young = pte_young(*pte);
> > > > > + if (!young)
> > > > > + young = !page_is_idle(pte_page(*pte));
> > > > > + pte_unmap_unlock(pte, ptl);
> > > > > + return young;
> > > > > + }
> > > > > +
> > > > > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> > > > > + young = pmd_young(*pmd);
> > > > > + if (!young)
> > > > > + young = !page_is_idle(pmd_page(*pmd));
> > > > > + spin_unlock(ptl);
> > > > > + *page_sz = ((1UL) << HPAGE_PMD_SHIFT);
> > > > > +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> > > > > +
> > > > > + return young;
> > > >
> > > > You need mmu_notifier_test_young() here. Hmm I remember mentioning
> > > > this in some previous version as well.
> > >
> > > Your question and my answer was as below:
> > >
> > > > Don't you need mmu_notifier_clear_young() here?
> > >
> > > I think we don't need it here because we only read the Accessed bit and PG_Idle
> > > if Accessed bit was not set.
> > >
> > > I should notice that you mean 'test_young()' but didn't, sorry. I will add it
> > > in the next version.
> > >
> >
> > I should have said mmu_notifier_test_young() instead of
> > mmu_notifier_clear_young().
> >
> > > >
> > > > BTW have you tested this on a VM?
> > >
> > > Yes. Indeed, I'm testing this on a QEMU/KVM environment. You can get more
> > > detail at: https://damonitor.github.io/doc/html/latest/vm/damon/eval.html#setup
> > >
> >
> > Hmm without mmu_notifier_test_young() you should be missing the kvm
> > mmu access updates. Can you please recheck if your eval is correctly
> > seeing the memory accesses from the VM?
>
> Seems I didn't clearly answered, sorry. My test setup installs the
> DAMON-enabled kernel in a guest VM and run it for workloads in the guest,
> rather than running DAMON in host to monitor accesses of VMs. The MMU notifier
> is for latter case, AFAIU, so my test setup didn't see the problem.
Just FYI. I confirmed the mmu_notifier_test_young() added version works for
the use case. I tested it by running a program accessing 200MB memory in a
QEMU/KVM guest having 120GB memory and monitoring the qemu process' virtual
address space from the host using DAMON. The 200MB memory region was clearly
identifiable.
Thanks,
SeongJae Park
On Tue, Dec 15, 2020 at 3:59 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> DAMON is designed to be used by kernel space code such as the memory
> management subsystems, and therefore it provides only kernel space API.
Which kernel space APIs are being referred here?
> That said, letting the user space control DAMON could provide some
> benefits to them. For example, it will allow user space to analyze
> their specific workloads and make their own special optimizations.
>
> For such cases, this commit implements a simple DAMON application kernel
> module, namely 'damon-dbgfs', which merely wraps the DAMON api and
> exports those to the user space via the debugfs.
>
> 'damon-dbgfs' exports three files, ``attrs``, ``target_ids``, and
> ``monitor_on`` under its debugfs directory, ``<debugfs>/damon/``.
>
> Attributes
> ----------
>
> Users can read and write the ``sampling interval``, ``aggregation
> interval``, ``regions update interval``, and min/max number of
> monitoring target regions by reading from and writing to the ``attrs``
> file. For example, below commands set those values to 5 ms, 100 ms,
> 1,000 ms, 10, 1000 and check it again::
>
> # cd <debugfs>/damon
> # echo 5000 100000 1000000 10 1000 > attrs
> # cat attrs
> 5000 100000 1000000 10 1000
>
> Target IDs
> ----------
>
> Some types of address spaces supports multiple monitoring target. For
> example, the virtual memory address spaces monitoring can have multiple
> processes as the monitoring targets. Users can set the targets by
> writing relevant id values of the targets to, and get the ids of the
> current targets by reading from the ``target_ids`` file. In case of the
> virtual address spaces monitoring, the values should be pids of the
> monitoring target processes. For example, below commands set processes
> having pids 42 and 4242 as the monitoring targets and check it again::
>
> # cd <debugfs>/damon
> # echo 42 4242 > target_ids
> # cat target_ids
> 42 4242
>
> Note that setting the target ids doesn't start the monitoring.
>
> Turning On/Off
> --------------
>
> Setting the files as described above doesn't incur effect unless you
> explicitly start the monitoring. You can start, stop, and check the
> current status of the monitoring by writing to and reading from the
> ``monitor_on`` file. Writing ``on`` to the file starts the monitoring
> of the targets with the attributes. Writing ``off`` to the file stops
> those. DAMON also stops if every targets are invalidated (in case of
> the virtual memory monitoring, target processes are invalidated when
> terminated). Below example commands turn on, off, and check the status
> of DAMON::
>
> # cd <debugfs>/damon
> # echo on > monitor_on
> # echo off > monitor_on
> # cat monitor_on
> off
>
> Please note that you cannot write to the above-mentioned debugfs files
> while the monitoring is turned on. If you write to the files while
> DAMON is running, an error code such as ``-EBUSY`` will be returned.
>
> Signed-off-by: SeongJae Park <[email protected]>
> Reviewed-by: Leonard Foerster <[email protected]>
> ---
> include/linux/damon.h | 3 +
> mm/damon/Kconfig | 9 ++
> mm/damon/Makefile | 1 +
> mm/damon/core.c | 45 ++++++
> mm/damon/dbgfs.c | 366 ++++++++++++++++++++++++++++++++++++++++++
> 5 files changed, 424 insertions(+)
> create mode 100644 mm/damon/dbgfs.c
>
> diff --git a/include/linux/damon.h b/include/linux/damon.h
> index 39b4d6d3ddee..f9e0d4349352 100644
> --- a/include/linux/damon.h
> +++ b/include/linux/damon.h
> @@ -265,9 +265,12 @@ unsigned int damon_nr_regions(struct damon_target *t);
>
> struct damon_ctx *damon_new_ctx(enum damon_target_type type);
> void damon_destroy_ctx(struct damon_ctx *ctx);
> +int damon_set_targets(struct damon_ctx *ctx,
> + unsigned long *ids, ssize_t nr_ids);
> int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> unsigned long aggr_int, unsigned long regions_update_int,
> unsigned long min_nr_reg, unsigned long max_nr_reg);
> +int damon_nr_running_ctxs(void);
>
> int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
> int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
> diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
> index 8ae080c52950..72f1683ba0ee 100644
> --- a/mm/damon/Kconfig
> +++ b/mm/damon/Kconfig
> @@ -21,4 +21,13 @@ config DAMON_VADDR
> This builds the default data access monitoring primitives for DAMON
> that works for virtual address spaces.
>
> +config DAMON_DBGFS
> + bool "DAMON debugfs interface"
> + depends on DAMON_VADDR && DEBUG_FS
> + help
> + This builds the debugfs interface for DAMON. The user space admins
> + can use the interface for arbitrary data access monitoring.
> +
> + If unsure, say N.
> +
> endmenu
> diff --git a/mm/damon/Makefile b/mm/damon/Makefile
> index 6ebbd08aed67..fed4be3bace3 100644
> --- a/mm/damon/Makefile
> +++ b/mm/damon/Makefile
> @@ -2,3 +2,4 @@
>
> obj-$(CONFIG_DAMON) := core.o
> obj-$(CONFIG_DAMON_VADDR) += vaddr.o
> +obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
> diff --git a/mm/damon/core.c b/mm/damon/core.c
> index 5ca9f79ccbb6..b9575a6bebff 100644
> --- a/mm/damon/core.c
> +++ b/mm/damon/core.c
> @@ -166,6 +166,37 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
> kfree(ctx);
> }
>
> +/**
> + * damon_set_targets() - Set monitoring targets.
> + * @ctx: monitoring context
> + * @ids: array of target ids
> + * @nr_ids: number of entries in @ids
> + *
> + * This function should not be called while the kdamond is running.
> + *
> + * Return: 0 on success, negative error code otherwise.
> + */
> +int damon_set_targets(struct damon_ctx *ctx,
> + unsigned long *ids, ssize_t nr_ids)
> +{
> + ssize_t i;
> + struct damon_target *t, *next;
> +
> + damon_for_each_target_safe(t, next, ctx)
> + damon_destroy_target(t);
You need to put the reference on the target before destroying.
> +
> + for (i = 0; i < nr_ids; i++) {
> + t = damon_new_target(ids[i]);
> + if (!t) {
> + pr_err("Failed to alloc damon_target\n");
> + return -ENOMEM;
> + }
> + damon_add_target(ctx, t);
> + }
> +
> + return 0;
> +}
> +
> /**
> * damon_set_attrs() - Set attributes for the monitoring.
> * @ctx: monitoring context
> @@ -206,6 +237,20 @@ int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> return 0;
> }
>
> +/**
> + * damon_nr_running_ctxs() - Return number of currently running contexts.
> + */
> +int damon_nr_running_ctxs(void)
> +{
> + int nr_ctxs;
> +
> + mutex_lock(&damon_lock);
> + nr_ctxs = nr_running_ctxs;
> + mutex_unlock(&damon_lock);
READ_ONCE(nr_running_ctxs) ?
> +
> + return nr_ctxs;
> +}
> +
> /* Returns the size upper limit for each monitoring region */
> static unsigned long damon_region_sz_limit(struct damon_ctx *ctx)
> {
> diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
> new file mode 100644
> index 000000000000..fd1665a183c2
> --- /dev/null
> +++ b/mm/damon/dbgfs.c
> @@ -0,0 +1,366 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * DAMON Debugfs Interface
> + *
> + * Author: SeongJae Park <[email protected]>
> + */
> +
> +#define pr_fmt(fmt) "damon-dbgfs: " fmt
> +
> +#include <linux/damon.h>
> +#include <linux/debugfs.h>
> +#include <linux/file.h>
> +#include <linux/mm.h>
> +#include <linux/module.h>
> +#include <linux/page_idle.h>
> +#include <linux/slab.h>
> +
> +static struct damon_ctx **dbgfs_ctxs;
> +static int dbgfs_nr_ctxs;
> +static struct dentry **dbgfs_dirs;
> +
> +/*
> + * Returns non-empty string on success, negarive error code otherwise.
> + */
> +static char *user_input_str(const char __user *buf, size_t count, loff_t *ppos)
> +{
> + char *kbuf;
> + ssize_t ret;
> +
> + /* We do not accept continuous write */
> + if (*ppos)
> + return ERR_PTR(-EINVAL);
> +
> + kbuf = kmalloc(count + 1, GFP_KERNEL);
> + if (!kbuf)
> + return ERR_PTR(-ENOMEM);
> +
> + ret = simple_write_to_buffer(kbuf, count + 1, ppos, buf, count);
> + if (ret != count) {
> + kfree(kbuf);
> + return ERR_PTR(-EIO);
> + }
> + kbuf[ret] = '\0';
> +
> + return kbuf;
> +}
> +
> +static ssize_t dbgfs_attrs_read(struct file *file,
> + char __user *buf, size_t count, loff_t *ppos)
> +{
> + struct damon_ctx *ctx = file->private_data;
> + char kbuf[128];
> + int ret;
> +
> + mutex_lock(&ctx->kdamond_lock);
> + ret = scnprintf(kbuf, ARRAY_SIZE(kbuf), "%lu %lu %lu %lu %lu\n",
> + ctx->sample_interval, ctx->aggr_interval,
> + ctx->regions_update_interval, ctx->min_nr_regions,
> + ctx->max_nr_regions);
> + mutex_unlock(&ctx->kdamond_lock);
> +
> + return simple_read_from_buffer(buf, count, ppos, kbuf, ret);
> +}
> +
> +static ssize_t dbgfs_attrs_write(struct file *file,
> + const char __user *buf, size_t count, loff_t *ppos)
> +{
> + struct damon_ctx *ctx = file->private_data;
> + unsigned long s, a, r, minr, maxr;
> + char *kbuf;
> + ssize_t ret = count;
> + int err;
> +
> + kbuf = user_input_str(buf, count, ppos);
> + if (IS_ERR(kbuf))
> + return PTR_ERR(kbuf);
> +
> + if (sscanf(kbuf, "%lu %lu %lu %lu %lu",
> + &s, &a, &r, &minr, &maxr) != 5) {
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + mutex_lock(&ctx->kdamond_lock);
> + if (ctx->kdamond) {
> + ret = -EBUSY;
> + goto unlock_out;
> + }
> +
> + err = damon_set_attrs(ctx, s, a, r, minr, maxr);
> + if (err)
> + ret = err;
> +unlock_out:
> + mutex_unlock(&ctx->kdamond_lock);
> +out:
> + kfree(kbuf);
> + return ret;
> +}
> +
> +#define targetid_is_pid(ctx) \
> + (ctx->primitive.target_valid == damon_va_target_valid)
> +
> +static ssize_t sprint_target_ids(struct damon_ctx *ctx, char *buf, ssize_t len)
> +{
> + struct damon_target *t;
> + unsigned long id;
> + int written = 0;
> + int rc;
> +
> + damon_for_each_target(t, ctx) {
> + id = t->id;
> + if (targetid_is_pid(ctx))
> + /* Show pid numbers to debugfs users */
> + id = (unsigned long)pid_vnr((struct pid *)id);
> +
> + rc = scnprintf(&buf[written], len - written, "%lu ", id);
> + if (!rc)
> + return -ENOMEM;
> + written += rc;
> + }
> + if (written)
> + written -= 1;
> + written += scnprintf(&buf[written], len - written, "\n");
> + return written;
> +}
> +
> +static ssize_t dbgfs_target_ids_read(struct file *file,
> + char __user *buf, size_t count, loff_t *ppos)
> +{
> + struct damon_ctx *ctx = file->private_data;
> + ssize_t len;
> + char ids_buf[320];
> +
> + mutex_lock(&ctx->kdamond_lock);
> + len = sprint_target_ids(ctx, ids_buf, 320);
> + mutex_unlock(&ctx->kdamond_lock);
> + if (len < 0)
> + return len;
> +
> + return simple_read_from_buffer(buf, count, ppos, ids_buf, len);
> +}
> +
> +/*
> + * Converts a string into an array of unsigned long integers
> + *
> + * Returns an array of unsigned long integers if the conversion success, or
> + * NULL otherwise.
> + */
> +static unsigned long *str_to_target_ids(const char *str, ssize_t len,
> + ssize_t *nr_ids)
> +{
> + unsigned long *ids;
> + const int max_nr_ids = 32;
> + unsigned long id;
> + int pos = 0, parsed, ret;
> +
> + *nr_ids = 0;
> + ids = kmalloc_array(max_nr_ids, sizeof(id), GFP_KERNEL);
> + if (!ids)
> + return NULL;
> + while (*nr_ids < max_nr_ids && pos < len) {
> + ret = sscanf(&str[pos], "%lu%n", &id, &parsed);
> + pos += parsed;
> + if (ret != 1)
> + break;
> + ids[*nr_ids] = id;
> + *nr_ids += 1;
> + }
> +
> + return ids;
> +}
> +
> +static ssize_t dbgfs_target_ids_write(struct file *file,
> + const char __user *buf, size_t count, loff_t *ppos)
> +{
> + struct damon_ctx *ctx = file->private_data;
> + char *kbuf, *nrs;
> + unsigned long *targets;
> + ssize_t nr_targets;
> + ssize_t ret = count;
> + int i;
> + int err;
> +
> + kbuf = user_input_str(buf, count, ppos);
> + if (IS_ERR(kbuf))
> + return PTR_ERR(kbuf);
> +
> + nrs = kbuf;
> +
> + targets = str_to_target_ids(nrs, ret, &nr_targets);
> + if (!targets) {
> + ret = -ENOMEM;
> + goto out;
> + }
> +
> + if (targetid_is_pid(ctx)) {
> + for (i = 0; i < nr_targets; i++)
> + targets[i] = (unsigned long)find_get_pid(
> + (int)targets[i]);
> + }
> +
> + mutex_lock(&ctx->kdamond_lock);
> + if (ctx->kdamond) {
> + ret = -EINVAL;
> + goto unlock_out;
You need to put_pid on the targets array.
> + }
> +
> + err = damon_set_targets(ctx, targets, nr_targets);
> + if (err)
> + ret = err;
You need to handle the partial failure from damon_set_targets().
> +unlock_out:
> + mutex_unlock(&ctx->kdamond_lock);
> + kfree(targets);
> +out:
> + kfree(kbuf);
> + return ret;
> +}
> +
> +static int damon_dbgfs_open(struct inode *inode, struct file *file)
> +{
> + file->private_data = inode->i_private;
> +
> + return nonseekable_open(inode, file);
> +}
> +
> +static const struct file_operations attrs_fops = {
> + .owner = THIS_MODULE,
> + .open = damon_dbgfs_open,
> + .read = dbgfs_attrs_read,
> + .write = dbgfs_attrs_write,
> +};
> +
> +static const struct file_operations target_ids_fops = {
> + .owner = THIS_MODULE,
> + .open = damon_dbgfs_open,
> + .read = dbgfs_target_ids_read,
> + .write = dbgfs_target_ids_write,
> +};
> +
> +static int dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx)
> +{
> + const char * const file_names[] = {"attrs", "target_ids"};
> + const struct file_operations *fops[] = {&attrs_fops, &target_ids_fops};
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(file_names); i++) {
> + if (!debugfs_create_file(file_names[i], 0600, dir,
> + ctx, fops[i])) {
> + pr_err("failed to create %s file\n", file_names[i]);
> + return -ENOMEM;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static struct damon_ctx *dbgfs_new_ctx(void)
> +{
> + struct damon_ctx *ctx;
> +
> + ctx = damon_new_ctx(DAMON_ADAPTIVE_TARGET);
> + if (!ctx)
> + return NULL;
> +
> + damon_va_set_primitives(ctx);
> + return ctx;
> +}
> +
> +static ssize_t dbgfs_monitor_on_read(struct file *file,
> + char __user *buf, size_t count, loff_t *ppos)
> +{
> + char monitor_on_buf[5];
> + bool monitor_on = damon_nr_running_ctxs() != 0;
> + int len;
> +
> + len = scnprintf(monitor_on_buf, 5, monitor_on ? "on\n" : "off\n");
> +
> + return simple_read_from_buffer(buf, count, ppos, monitor_on_buf, len);
> +}
> +
> +static ssize_t dbgfs_monitor_on_write(struct file *file,
> + const char __user *buf, size_t count, loff_t *ppos)
> +{
> + ssize_t ret = count;
> + char *kbuf;
> + int err;
> +
> + kbuf = user_input_str(buf, count, ppos);
> + if (IS_ERR(kbuf))
> + return PTR_ERR(kbuf);
> +
> + /* Remove white space */
> + if (sscanf(kbuf, "%s", kbuf) != 1) {
> + kfree(kbuf);
> + return -EINVAL;
> + }
> +
> + if (!strncmp(kbuf, "on", count))
> + err = damon_start(dbgfs_ctxs, dbgfs_nr_ctxs);
> + else if (!strncmp(kbuf, "off", count))
> + err = damon_stop(dbgfs_ctxs, dbgfs_nr_ctxs);
> + else
> + err = -EINVAL;
> +
> + if (err)
> + ret = err;
> + kfree(kbuf);
> + return ret;
> +}
> +
> +static const struct file_operations monitor_on_fops = {
> + .owner = THIS_MODULE,
> + .read = dbgfs_monitor_on_read,
> + .write = dbgfs_monitor_on_write,
> +};
> +
> +static int __init __damon_dbgfs_init(void)
> +{
> + struct dentry *dbgfs_root;
> + const char * const file_names[] = {"monitor_on"};
> + const struct file_operations *fops[] = {&monitor_on_fops};
> + int i;
> +
> + dbgfs_root = debugfs_create_dir("damon", NULL);
> + if (IS_ERR(dbgfs_root)) {
> + pr_err("failed to create the dbgfs dir\n");
> + return PTR_ERR(dbgfs_root);
> + }
> +
> + for (i = 0; i < ARRAY_SIZE(file_names); i++) {
> + if (!debugfs_create_file(file_names[i], 0600, dbgfs_root,
> + NULL, fops[i])) {
> + pr_err("failed to create %s file\n", file_names[i]);
> + return -ENOMEM;
> + }
> + }
> + dbgfs_fill_ctx_dir(dbgfs_root, dbgfs_ctxs[0]);
> +
> + dbgfs_dirs = kmalloc_array(1, sizeof(dbgfs_root), GFP_KERNEL);
> + dbgfs_dirs[0] = dbgfs_root;
> +
> + return 0;
> +}
> +
> +/*
> + * Functions for the initialization
> + */
> +
> +static int __init damon_dbgfs_init(void)
> +{
> + int rc;
> +
> + dbgfs_ctxs = kmalloc(sizeof(*dbgfs_ctxs), GFP_KERNEL);
> + dbgfs_ctxs[0] = dbgfs_new_ctx();
> + if (!dbgfs_ctxs[0])
> + return -ENOMEM;
> + dbgfs_nr_ctxs = 1;
> +
> + rc = __damon_dbgfs_init();
> + if (rc)
> + pr_err("%s: dbgfs init failed\n", __func__);
> +
> + return rc;
> +}
> +
> +module_init(damon_dbgfs_init);
> --
> 2.17.1
>
On Tue, Dec 15, 2020 at 3:56 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> To avoid the unbounded increase of the overhead, DAMON groups adjacent
> pages that assumed to have the same access frequencies into a region.
'that are assumed'
> As long as the assumption (pages in a region have the same access
> frequencies) is kept, only one page in the region is required to be
> checked. Thus, for each ``sampling interval``,
>
> 1. the 'prepare_access_checks' primitive picks one page in each region,
> 2. waits for one ``sampling interval``,
> 3. checks whether the page is accessed meanwhile, and
> 4. increases the access frequency of the region if so.
I think you meant increasing the access 'count' or something.
Increasing the access frequency somewhat conveys that the sampling
interval is being decreased.
>
> Therefore, the monitoring overhead is controllable by adjusting the
> number of regions. DAMON allows both the underlying primitives and user
> callbacks adjust regions for the trade-off. In other words, this commit
'callbacks to adjust'
> makes DAMON to use not only time-based sampling but also space-based
> sampling.
>
> This scheme, however, cannot preserve the quality of the output if the
> assumption is not guaranteed. Next commit will address this problem.
>
> Another problem of this region abstraction is additional memory space
> overhead for the regions metadata. For example, suppose page
> granularity monitoring that doesn't want to know fine-grained access
> frequency but only if each page accessed or not.
You mean when the sampling interval is equal to the aggregation interval, right?
> Then, we can do that
> by directly resetting and reading the PG_Idle flags and/or the PTE
> Accessed bits. The metadata for the region abstraction is only burden
> in the case. For the reason, this commit makes DAMON to support the
> user-defined arbitrary target, which could be stored in a void pointer
> of the monitoring context with specific target type.
Sorry I didn't follow. How does sampling interval equal to aggregation
interval require user-defined arbitrary targets?
>
> Signed-off-by: SeongJae Park <[email protected]>
> Reviewed-by: Leonard Foerster <[email protected]>
> ---
> include/linux/damon.h | 109 ++++++++++++++++++++++++++++++--
> mm/damon/core.c | 142 +++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 243 insertions(+), 8 deletions(-)
>
> diff --git a/include/linux/damon.h b/include/linux/damon.h
> index 387fa4399fc8..7d4685adc8a9 100644
> --- a/include/linux/damon.h
> +++ b/include/linux/damon.h
> @@ -12,6 +12,48 @@
> #include <linux/time64.h>
> #include <linux/types.h>
>
> +/**
> + * struct damon_addr_range - Represents an address region of [@start, @end).
> + * @start: Start address of the region (inclusive).
> + * @end: End address of the region (exclusive).
> + */
> +struct damon_addr_range {
> + unsigned long start;
> + unsigned long end;
> +};
> +
> +/**
> + * struct damon_region - Represents a monitoring target region.
> + * @ar: The address range of the region.
> + * @sampling_addr: Address of the sample for the next access check.
> + * @nr_accesses: Access frequency of this region.
> + * @list: List head for siblings.
> + */
> +struct damon_region {
> + struct damon_addr_range ar;
> + unsigned long sampling_addr;
> + unsigned int nr_accesses;
> + struct list_head list;
> +};
> +
> +/**
> + * struct damon_target - Represents a monitoring target.
> + * @id: Unique identifier for this target.
> + * @regions_list: Head of the monitoring target regions of this target.
> + * @list: List head for siblings.
> + *
> + * Each monitoring context could have multiple targets. For example, a context
> + * for virtual memory address spaces could have multiple target processes. The
> + * @id of each target should be unique among the targets of the context. For
> + * example, in the virtual address monitoring context, it could be a pidfd or
> + * an address of an mm_struct.
> + */
> +struct damon_target {
> + unsigned long id;
> + struct list_head regions_list;
> + struct list_head list;
> +};
> +
> struct damon_ctx;
>
> /**
> @@ -36,7 +78,8 @@ struct damon_ctx;
> *
> * @init_target_regions should construct proper monitoring target regions and
> * link those to the DAMON context struct. The regions should be defined by
> - * user and saved in @damon_ctx.target.
> + * user and saved in @damon_ctx.arbitrary_target if @damon_ctx.target_type is
> + * &DAMON_ARBITRARY_TARGET. Otherwise, &struct damon_region should be used.
> * @update_target_regions should update the monitoring target regions for
> * current status.
> * @prepare_access_checks should manipulate the monitoring regions to be
> @@ -46,7 +89,8 @@ struct damon_ctx;
> * @reset_aggregated should reset the access monitoring results that aggregated
> * by @check_accesses.
> * @target_valid should check whether the target is still valid for the
> - * monitoring.
> + * monitoring. It receives &damon_ctx.arbitrary_target or &struct damon_target
> + * pointer depends on &damon_ctx.target_type.
> * @cleanup is called from @kdamond just before its termination. After this
> * call, only @kdamond_lock and @kdamond will be touched.
> */
> @@ -91,6 +135,17 @@ struct damon_callback {
> int (*before_terminate)(struct damon_ctx *context);
> };
>
> +/**
> + * enum damon_target_type - Represents the type of the monitoring target.
> + *
> + * @DAMON_REGION_SAMPLING_TARGET: Region based sampling target.
> + * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
> + */
> +enum damon_target_type {
> + DAMON_REGION_SAMPLING_TARGET,
> + DAMON_ARBITRARY_TARGET,
I would suggest removing the arbitrary target in this pathset. This
patchset is only adding the region sampling target, so no need to add
the arbitrary target here.
> +};
> +
> /**
> * struct damon_ctx - Represents a context for each monitoring. This is the
> * main interface that allows users to set the attributes and get the results
> @@ -130,7 +185,15 @@ struct damon_callback {
> * @primitive: Set of monitoring primitives for given use cases.
> * @callback: Set of callbacks for monitoring events notifications.
> *
> - * @target: Pointer to the user-defined monitoring target.
> + * @target_type: Type of the monitoring target.
> + *
> + * @region_targets: Head of monitoring targets (&damon_target) list.
> + *
> + * @arbitrary_target: Pointer to arbitrary type target.
> + *
> + * @region_targets are valid only if @target_type is
> + * DAMON_REGION_SAMPLING_TARGET. @arbitrary_target is valid only if
> + * @target_type is DAMON_ARBITRARY_TARGET.
> */
> struct damon_ctx {
> unsigned long sample_interval;
> @@ -149,12 +212,48 @@ struct damon_ctx {
> struct damon_primitive primitive;
> struct damon_callback callback;
>
> - void *target;
> + enum damon_target_type target_type;
> + union {
> + /* DAMON_REGION_SAMPLING_TARGET */
> + struct list_head region_targets;
> +
> + /* DAMON_ARBITRARY_TARGET */
> + void *arbitrary_target;
> + };
> };
>
> +#define damon_next_region(r) \
> + (container_of(r->list.next, struct damon_region, list))
> +
> +#define damon_prev_region(r) \
> + (container_of(r->list.prev, struct damon_region, list))
> +
> +#define damon_for_each_region(r, t) \
> + list_for_each_entry(r, &t->regions_list, list)
> +
> +#define damon_for_each_region_safe(r, next, t) \
> + list_for_each_entry_safe(r, next, &t->regions_list, list)
> +
> +#define damon_for_each_target(t, ctx) \
> + list_for_each_entry(t, &(ctx)->region_targets, list)
> +
> +#define damon_for_each_target_safe(t, next, ctx) \
> + list_for_each_entry_safe(t, next, &(ctx)->region_targets, list)
> +
> #ifdef CONFIG_DAMON
>
> -struct damon_ctx *damon_new_ctx(void);
> +struct damon_region *damon_new_region(unsigned long start, unsigned long end);
> +inline void damon_insert_region(struct damon_region *r,
> + struct damon_region *prev, struct damon_region *next);
> +void damon_add_region(struct damon_region *r, struct damon_target *t);
> +void damon_destroy_region(struct damon_region *r);
> +
> +struct damon_target *damon_new_target(unsigned long id);
> +void damon_add_target(struct damon_ctx *ctx, struct damon_target *t);
> +void damon_free_target(struct damon_target *t);
> +void damon_destroy_target(struct damon_target *t);
> +
> +struct damon_ctx *damon_new_ctx(enum damon_target_type type);
> void damon_destroy_ctx(struct damon_ctx *ctx);
> int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> unsigned long aggr_int, unsigned long regions_update_int);
> diff --git a/mm/damon/core.c b/mm/damon/core.c
> index 8963804efdf9..167487e75737 100644
> --- a/mm/damon/core.c
> +++ b/mm/damon/core.c
> @@ -15,7 +15,102 @@
> static DEFINE_MUTEX(damon_lock);
> static int nr_running_ctxs;
>
> -struct damon_ctx *damon_new_ctx(void)
> +/*
> + * Construct a damon_region struct
> + *
> + * Returns the pointer to the new struct if success, or NULL otherwise
> + */
> +struct damon_region *damon_new_region(unsigned long start, unsigned long end)
> +{
> + struct damon_region *region;
> +
> + region = kmalloc(sizeof(*region), GFP_KERNEL);
> + if (!region)
> + return NULL;
> +
> + region->ar.start = start;
> + region->ar.end = end;
> + region->nr_accesses = 0;
> + INIT_LIST_HEAD(®ion->list);
> +
> + return region;
> +}
> +
> +/*
> + * Add a region between two other regions
> + */
> +inline void damon_insert_region(struct damon_region *r,
> + struct damon_region *prev, struct damon_region *next)
> +{
> + __list_add(&r->list, &prev->list, &next->list);
> +}
> +
> +void damon_add_region(struct damon_region *r, struct damon_target *t)
> +{
> + list_add_tail(&r->list, &t->regions_list);
> +}
> +
I don't see the benefit of these one line functions at least the following two.
> +static void damon_del_region(struct damon_region *r)
> +{
> + list_del(&r->list);
> +}
> +
> +static void damon_free_region(struct damon_region *r)
> +{
> + kfree(r);
> +}
> +
> +void damon_destroy_region(struct damon_region *r)
> +{
> + damon_del_region(r);
> + damon_free_region(r);
> +}
> +
> +/*
> + * Construct a damon_target struct
> + *
> + * Returns the pointer to the new struct if success, or NULL otherwise
> + */
> +struct damon_target *damon_new_target(unsigned long id)
> +{
> + struct damon_target *t;
> +
> + t = kmalloc(sizeof(*t), GFP_KERNEL);
> + if (!t)
> + return NULL;
> +
> + t->id = id;
> + INIT_LIST_HEAD(&t->regions_list);
> +
> + return t;
> +}
> +
> +void damon_add_target(struct damon_ctx *ctx, struct damon_target *t)
> +{
> + list_add_tail(&t->list, &ctx->region_targets);
> +}
> +
> +static void damon_del_target(struct damon_target *t)
> +{
> + list_del(&t->list);
> +}
> +
> +void damon_free_target(struct damon_target *t)
> +{
> + struct damon_region *r, *next;
> +
> + damon_for_each_region_safe(r, next, t)
> + damon_free_region(r);
> + kfree(t);
> +}
> +
> +void damon_destroy_target(struct damon_target *t)
> +{
> + damon_del_target(t);
> + damon_free_target(t);
> +}
> +
> +struct damon_ctx *damon_new_ctx(enum damon_target_type type)
> {
> struct damon_ctx *ctx;
>
> @@ -32,13 +127,20 @@ struct damon_ctx *damon_new_ctx(void)
>
> mutex_init(&ctx->kdamond_lock);
>
> - ctx->target = NULL;
> + ctx->target_type = type;
> + if (type != DAMON_ARBITRARY_TARGET)
> + INIT_LIST_HEAD(&ctx->region_targets);
>
> return ctx;
> }
>
> void damon_destroy_ctx(struct damon_ctx *ctx)
> {
> + struct damon_target *t, *next_t;
> +
> + damon_for_each_target_safe(t, next_t, ctx)
> + damon_destroy_target(t);
> +
> kfree(ctx);
> }
>
> @@ -215,6 +317,21 @@ static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx)
> ctx->aggr_interval);
> }
>
> +/*
> + * Reset the aggregated monitoring results ('nr_accesses' of each region).
> + */
> +static void kdamond_reset_aggregated(struct damon_ctx *c)
> +{
> + struct damon_target *t;
> +
> + damon_for_each_target(t, c) {
> + struct damon_region *r;
> +
> + damon_for_each_region(r, t)
> + r->nr_accesses = 0;
> + }
> +}
> +
> /*
> * Check whether it is time to check and apply the target monitoring regions
> *
> @@ -236,6 +353,7 @@ static bool kdamond_need_update_regions(struct damon_ctx *ctx)
> */
> static bool kdamond_need_stop(struct damon_ctx *ctx)
> {
> + struct damon_target *t;
> bool stop;
>
> mutex_lock(&ctx->kdamond_lock);
> @@ -247,7 +365,15 @@ static bool kdamond_need_stop(struct damon_ctx *ctx)
> if (!ctx->primitive.target_valid)
> return false;
>
> - return !ctx->primitive.target_valid(ctx->target);
> + if (ctx->target_type == DAMON_ARBITRARY_TARGET)
> + return !ctx->primitive.target_valid(ctx->arbitrary_target);
> +
> + damon_for_each_target(t, ctx) {
> + if (ctx->primitive.target_valid(t))
> + return false;
> + }
> +
> + return true;
> }
>
> static void set_kdamond_stop(struct damon_ctx *ctx)
> @@ -263,6 +389,8 @@ static void set_kdamond_stop(struct damon_ctx *ctx)
> static int kdamond_fn(void *data)
> {
> struct damon_ctx *ctx = (struct damon_ctx *)data;
> + struct damon_target *t;
> + struct damon_region *r, *next;
>
> pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
>
> @@ -287,6 +415,8 @@ static int kdamond_fn(void *data)
> if (ctx->callback.after_aggregation &&
> ctx->callback.after_aggregation(ctx))
> set_kdamond_stop(ctx);
> + if (ctx->target_type != DAMON_ARBITRARY_TARGET)
> + kdamond_reset_aggregated(ctx);
> if (ctx->primitive.reset_aggregated)
> ctx->primitive.reset_aggregated(ctx);
> }
> @@ -296,6 +426,12 @@ static int kdamond_fn(void *data)
> ctx->primitive.update_target_regions(ctx);
> }
> }
> + if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
> + damon_for_each_target(t, ctx) {
> + damon_for_each_region_safe(r, next, t)
> + damon_destroy_region(r);
> + }
> + }
>
> if (ctx->callback.before_terminate &&
> ctx->callback.before_terminate(ctx))
> --
> 2.17.1
>
On Tue, Dec 15, 2020 at 3:57 AM SeongJae Park <[email protected]> wrote:
>
> From: SeongJae Park <[email protected]>
>
> Even somehow the initial monitoring target regions are well constructed
> to fulfill the assumption (pages in same region have similar access
> frequencies), the data access pattern can be dynamically changed. This
> will result in low monitoring quality. To keep the assumption as much
> as possible, DAMON adaptively merges and splits each region based on
> their access frequency.
>
> For each ``aggregation interval``, it compares the access frequencies of
> adjacent regions and merges those if the frequency difference is small.
> Then, after it reports and clears the aggregated access frequency of
> each region, it splits each region into two or three regions if the
> total number of regions will not exceed the user-specified maximum
> number of regions after the split.
Should there be any concerns regarding the number of regions
oscillating even when the access pattern of the application is not
changing? Does the system converge to equilibrium state or does it not
matter?
>
> In this way, DAMON provides its best-effort quality and minimal overhead
> while keeping the upper-bound overhead that users set.
>
> Signed-off-by: SeongJae Park <[email protected]>
> Reviewed-by: Leonard Foerster <[email protected]>
> ---
> include/linux/damon.h | 41 +++++---
> mm/damon/core.c | 220 ++++++++++++++++++++++++++++++++++++++++--
> 2 files changed, 240 insertions(+), 21 deletions(-)
>
> diff --git a/include/linux/damon.h b/include/linux/damon.h
> index 7d4685adc8a9..f446f8433599 100644
> --- a/include/linux/damon.h
> +++ b/include/linux/damon.h
> @@ -12,6 +12,9 @@
> #include <linux/time64.h>
> #include <linux/types.h>
>
> +/* Minimal region size. Every damon_region is aligned by this. */
> +#define DAMON_MIN_REGION PAGE_SIZE
> +
> /**
> * struct damon_addr_range - Represents an address region of [@start, @end).
> * @start: Start address of the region (inclusive).
> @@ -86,6 +89,8 @@ struct damon_ctx;
> * prepared for the next access check.
> * @check_accesses should check the accesses to each region that made after the
> * last preparation and update the number of observed accesses of each region.
> + * It should also return max number of observed accesses that made as a result
> + * of its update.
Why?
> * @reset_aggregated should reset the access monitoring results that aggregated
> * by @check_accesses.
> * @target_valid should check whether the target is still valid for the
> @@ -98,7 +103,7 @@ struct damon_primitive {
> void (*init_target_regions)(struct damon_ctx *context);
> void (*update_target_regions)(struct damon_ctx *context);
> void (*prepare_access_checks)(struct damon_ctx *context);
> - void (*check_accesses)(struct damon_ctx *context);
> + unsigned int (*check_accesses)(struct damon_ctx *context);
> void (*reset_aggregated)(struct damon_ctx *context);
> bool (*target_valid)(void *target);
> void (*cleanup)(struct damon_ctx *context);
> @@ -138,11 +143,11 @@ struct damon_callback {
> /**
> * enum damon_target_type - Represents the type of the monitoring target.
> *
> - * @DAMON_REGION_SAMPLING_TARGET: Region based sampling target.
> - * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
> + * @DAMON_ADAPTIVE_TARGET: Adaptive regions adjustment applied target.
> + * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
> */
> enum damon_target_type {
> - DAMON_REGION_SAMPLING_TARGET,
> + DAMON_ADAPTIVE_TARGET,
> DAMON_ARBITRARY_TARGET,
> };
>
> @@ -187,13 +192,15 @@ enum damon_target_type {
> *
> * @target_type: Type of the monitoring target.
> *
> - * @region_targets: Head of monitoring targets (&damon_target) list.
> + * @min_nr_regions: The minimum number of adaptive monitoring regions.
> + * @max_nr_regions: The maximum number of adaptive monitoring regions.
> + * @adaptive_targets: Head of monitoring targets (&damon_target) list.
> *
> * @arbitrary_target: Pointer to arbitrary type target.
> *
> - * @region_targets are valid only if @target_type is
> - * DAMON_REGION_SAMPLING_TARGET. @arbitrary_target is valid only if
> - * @target_type is DAMON_ARBITRARY_TARGET.
> + * @min_nr_regions, @max_nr_regions and @adaptive_targets are valid only if
> + * @target_type is &DAMON_ADAPTIVE_TARGET. @arbitrary_target is valid only if
> + * @target_type is &DAMON_ARBITRARY_TARGET.
> */
> struct damon_ctx {
> unsigned long sample_interval;
> @@ -214,11 +221,13 @@ struct damon_ctx {
>
> enum damon_target_type target_type;
> union {
> - /* DAMON_REGION_SAMPLING_TARGET */
> - struct list_head region_targets;
> + struct { /* DAMON_ADAPTIVE_TARGET */
> + unsigned long min_nr_regions;
> + unsigned long max_nr_regions;
> + struct list_head adaptive_targets;
> + };
>
> - /* DAMON_ARBITRARY_TARGET */
> - void *arbitrary_target;
> + void *arbitrary_target; /* DAMON_ARBITRARY_TARGET */
> };
> };
>
> @@ -235,10 +244,10 @@ struct damon_ctx {
> list_for_each_entry_safe(r, next, &t->regions_list, list)
>
> #define damon_for_each_target(t, ctx) \
> - list_for_each_entry(t, &(ctx)->region_targets, list)
> + list_for_each_entry(t, &(ctx)->adaptive_targets, list)
>
> #define damon_for_each_target_safe(t, next, ctx) \
> - list_for_each_entry_safe(t, next, &(ctx)->region_targets, list)
> + list_for_each_entry_safe(t, next, &(ctx)->adaptive_targets, list)
>
> #ifdef CONFIG_DAMON
>
> @@ -252,11 +261,13 @@ struct damon_target *damon_new_target(unsigned long id);
> void damon_add_target(struct damon_ctx *ctx, struct damon_target *t);
> void damon_free_target(struct damon_target *t);
> void damon_destroy_target(struct damon_target *t);
> +unsigned int damon_nr_regions(struct damon_target *t);
>
> struct damon_ctx *damon_new_ctx(enum damon_target_type type);
> void damon_destroy_ctx(struct damon_ctx *ctx);
> int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> - unsigned long aggr_int, unsigned long regions_update_int);
> + unsigned long aggr_int, unsigned long regions_update_int,
> + unsigned long min_nr_reg, unsigned long max_nr_reg);
>
> int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
> int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
> diff --git a/mm/damon/core.c b/mm/damon/core.c
> index 167487e75737..0f9beb60d9dd 100644
> --- a/mm/damon/core.c
> +++ b/mm/damon/core.c
> @@ -10,8 +10,12 @@
> #include <linux/damon.h>
> #include <linux/delay.h>
> #include <linux/kthread.h>
> +#include <linux/random.h>
> #include <linux/slab.h>
>
> +/* Get a random number in [l, r) */
> +#define damon_rand(l, r) (l + prandom_u32_max(r - l))
> +
> static DEFINE_MUTEX(damon_lock);
> static int nr_running_ctxs;
>
> @@ -87,7 +91,7 @@ struct damon_target *damon_new_target(unsigned long id)
>
> void damon_add_target(struct damon_ctx *ctx, struct damon_target *t)
> {
> - list_add_tail(&t->list, &ctx->region_targets);
> + list_add_tail(&t->list, &ctx->adaptive_targets);
> }
>
> static void damon_del_target(struct damon_target *t)
> @@ -110,6 +114,17 @@ void damon_destroy_target(struct damon_target *t)
> damon_free_target(t);
> }
>
> +unsigned int damon_nr_regions(struct damon_target *t)
> +{
> + struct damon_region *r;
> + unsigned int nr_regions = 0;
> +
> + damon_for_each_region(r, t)
> + nr_regions++;
Why not just add the region_count filed in damon_target?
> +
> + return nr_regions;
> +}
> +
> struct damon_ctx *damon_new_ctx(enum damon_target_type type)
> {
> struct damon_ctx *ctx;
> @@ -128,8 +143,12 @@ struct damon_ctx *damon_new_ctx(enum damon_target_type type)
> mutex_init(&ctx->kdamond_lock);
>
> ctx->target_type = type;
> - if (type != DAMON_ARBITRARY_TARGET)
> - INIT_LIST_HEAD(&ctx->region_targets);
> + if (type != DAMON_ARBITRARY_TARGET) {
> + ctx->min_nr_regions = 10;
> + ctx->max_nr_regions = 1000;
IMO these settings/heuristics should be part of the virtual address
space monitor primitives and not be in the core monitor.
> +
> + INIT_LIST_HEAD(&ctx->adaptive_targets);
> + }
>
> return ctx;
> }
> @@ -150,6 +169,8 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
> * @sample_int: time interval between samplings
> * @aggr_int: time interval between aggregations
> * @regions_update_int: time interval between target regions update
> + * @min_nr_reg: minimal number of regions
> + * @max_nr_reg: maximum number of regions
> *
> * This function should not be called while the kdamond is running.
> * Every time interval is in micro-seconds.
> @@ -157,15 +178,51 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
> * Return: 0 on success, negative error code otherwise.
> */
> int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> - unsigned long aggr_int, unsigned long regions_update_int)
> + unsigned long aggr_int, unsigned long regions_update_int,
> + unsigned long min_nr_reg, unsigned long max_nr_reg)
> {
> + if (min_nr_reg < 3) {
> + pr_err("min_nr_regions (%lu) must be at least 3\n",
> + min_nr_reg);
> + return -EINVAL;
> + }
> + if (min_nr_reg > max_nr_reg) {
> + pr_err("invalid nr_regions. min (%lu) > max (%lu)\n",
> + min_nr_reg, max_nr_reg);
> + return -EINVAL;
> + }
> +
> ctx->sample_interval = sample_int;
> ctx->aggr_interval = aggr_int;
> ctx->regions_update_interval = regions_update_int;
> + if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
> + ctx->min_nr_regions = min_nr_reg;
> + ctx->max_nr_regions = max_nr_reg;
> + }
>
> return 0;
> }
>
> +/* Returns the size upper limit for each monitoring region */
> +static unsigned long damon_region_sz_limit(struct damon_ctx *ctx)
> +{
> + struct damon_target *t;
> + struct damon_region *r;
> + unsigned long sz = 0;
> +
> + damon_for_each_target(t, ctx) {
> + damon_for_each_region(r, t)
> + sz += r->ar.end - r->ar.start;
> + }
> +
> + if (ctx->min_nr_regions)
> + sz /= ctx->min_nr_regions;
> + if (sz < DAMON_MIN_REGION)
> + sz = DAMON_MIN_REGION;
> +
> + return sz;
> +}
> +
> static bool damon_kdamond_running(struct damon_ctx *ctx)
> {
> bool running;
> @@ -332,6 +389,146 @@ static void kdamond_reset_aggregated(struct damon_ctx *c)
> }
> }
>
> +#define sz_damon_region(r) (r->ar.end - r->ar.start)
> +
> +/*
> + * Merge two adjacent regions into one region
> + */
> +static void damon_merge_two_regions(struct damon_region *l,
> + struct damon_region *r)
> +{
> + unsigned long sz_l = sz_damon_region(l), sz_r = sz_damon_region(r);
> +
> + l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
> + (sz_l + sz_r);
> + l->ar.end = r->ar.end;
> + damon_destroy_region(r);
> +}
> +
> +#define diff_of(a, b) (a > b ? a - b : b - a)
> +
> +/*
> + * Merge adjacent regions having similar access frequencies
> + *
> + * t target affected by this merge operation
> + * thres '->nr_accesses' diff threshold for the merge
> + * sz_limit size upper limit of each region
> + */
> +static void damon_merge_regions_of(struct damon_target *t, unsigned int thres,
> + unsigned long sz_limit)
> +{
> + struct damon_region *r, *prev = NULL, *next;
> +
> + damon_for_each_region_safe(r, next, t) {
> + if (prev && prev->ar.end == r->ar.start &&
> + diff_of(prev->nr_accesses, r->nr_accesses) <= thres &&
> + sz_damon_region(prev) + sz_damon_region(r) <= sz_limit)
> + damon_merge_two_regions(prev, r);
> + else
> + prev = r;
> + }
> +}
> +
> +/*
> + * Merge adjacent regions having similar access frequencies
> + *
> + * threshold '->nr_accesses' diff threshold for the merge
> + * sz_limit size upper limit of each region
> + *
> + * This function merges monitoring target regions which are adjacent and their
> + * access frequencies are similar. This is for minimizing the monitoring
> + * overhead under the dynamically changeable access pattern. If a merge was
> + * unnecessarily made, later 'kdamond_split_regions()' will revert it.
> + */
> +static void kdamond_merge_regions(struct damon_ctx *c, unsigned int threshold,
> + unsigned long sz_limit)
> +{
> + struct damon_target *t;
> +
> + damon_for_each_target(t, c)
> + damon_merge_regions_of(t, threshold, sz_limit);
> +}
> +
> +/*
> + * Split a region in two
> + *
> + * r the region to be split
> + * sz_r size of the first sub-region that will be made
> + */
> +static void damon_split_region_at(struct damon_ctx *ctx,
> + struct damon_region *r, unsigned long sz_r)
> +{
> + struct damon_region *new;
> +
> + new = damon_new_region(r->ar.start + sz_r, r->ar.end);
> + r->ar.end = new->ar.start;
> +
> + damon_insert_region(new, r, damon_next_region(r));
> +}
> +
> +/* Split every region in the given target into 'nr_subs' regions */
> +static void damon_split_regions_of(struct damon_ctx *ctx,
> + struct damon_target *t, int nr_subs)
> +{
> + struct damon_region *r, *next;
> + unsigned long sz_region, sz_sub = 0;
> + int i;
> +
> + damon_for_each_region_safe(r, next, t) {
> + sz_region = r->ar.end - r->ar.start;
> +
> + for (i = 0; i < nr_subs - 1 &&
> + sz_region > 2 * DAMON_MIN_REGION; i++) {
> + /*
> + * Randomly select size of left sub-region to be at
> + * least 10 percent and at most 90% of original region
> + */
> + sz_sub = ALIGN_DOWN(damon_rand(1, 10) *
> + sz_region / 10, DAMON_MIN_REGION);
> + /* Do not allow blank region */
> + if (sz_sub == 0 || sz_sub >= sz_region)
> + continue;
> +
> + damon_split_region_at(ctx, r, sz_sub);
> + sz_region = sz_sub;
> + }
> + }
> +}
> +
> +/*
> + * Split every target region into randomly-sized small regions
> + *
> + * This function splits every target region into random-sized small regions if
> + * current total number of the regions is equal or smaller than half of the
> + * user-specified maximum number of regions. This is for maximizing the
> + * monitoring accuracy under the dynamically changeable access patterns. If a
> + * split was unnecessarily made, later 'kdamond_merge_regions()' will revert
> + * it.
> + */
> +static void kdamond_split_regions(struct damon_ctx *ctx)
> +{
> + struct damon_target *t;
> + unsigned int nr_regions = 0;
> + static unsigned int last_nr_regions;
> + int nr_subregions = 2;
> +
> + damon_for_each_target(t, ctx)
> + nr_regions += damon_nr_regions(t);
> +
> + if (nr_regions > ctx->max_nr_regions / 2)
> + return;
Shouldn't the limits on region be per-target instead of for the whole context?
> +
> + /* Maybe the middle of the region has different access frequency */
> + if (last_nr_regions == nr_regions &&
> + nr_regions < ctx->max_nr_regions / 3)
> + nr_subregions = 3;
> +
> + damon_for_each_target(t, ctx)
> + damon_split_regions_of(ctx, t, nr_subregions);
> +
> + last_nr_regions = nr_regions;
> +}
> +
> /*
> * Check whether it is time to check and apply the target monitoring regions
> *
> @@ -391,6 +588,8 @@ static int kdamond_fn(void *data)
> struct damon_ctx *ctx = (struct damon_ctx *)data;
> struct damon_target *t;
> struct damon_region *r, *next;
> + unsigned int max_nr_accesses = 0;
> + unsigned long sz_limit = 0;
>
> pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
>
> @@ -399,6 +598,8 @@ static int kdamond_fn(void *data)
> if (ctx->callback.before_start && ctx->callback.before_start(ctx))
> set_kdamond_stop(ctx);
>
> + sz_limit = damon_region_sz_limit(ctx);
> +
> while (!kdamond_need_stop(ctx)) {
> if (ctx->primitive.prepare_access_checks)
> ctx->primitive.prepare_access_checks(ctx);
> @@ -409,14 +610,20 @@ static int kdamond_fn(void *data)
> usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
>
> if (ctx->primitive.check_accesses)
> - ctx->primitive.check_accesses(ctx);
> + max_nr_accesses = ctx->primitive.check_accesses(ctx);
>
> if (kdamond_aggregate_interval_passed(ctx)) {
> + if (ctx->target_type != DAMON_ARBITRARY_TARGET)
> + kdamond_merge_regions(ctx,
> + max_nr_accesses / 10,
What's the reason behind this 10?
> + sz_limit);
> if (ctx->callback.after_aggregation &&
> ctx->callback.after_aggregation(ctx))
> set_kdamond_stop(ctx);
> - if (ctx->target_type != DAMON_ARBITRARY_TARGET)
> + if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
> kdamond_reset_aggregated(ctx);
> + kdamond_split_regions(ctx);
> + }
> if (ctx->primitive.reset_aggregated)
> ctx->primitive.reset_aggregated(ctx);
> }
> @@ -424,6 +631,7 @@ static int kdamond_fn(void *data)
> if (kdamond_need_update_regions(ctx)) {
> if (ctx->primitive.update_target_regions)
> ctx->primitive.update_target_regions(ctx);
> + sz_limit = damon_region_sz_limit(ctx);
> }
> }
> if (ctx->target_type != DAMON_ARBITRARY_TARGET) {
> --
> 2.17.1
>
On Mon, 1 Feb 2021 09:37:29 -0800 Shakeel Butt <[email protected]> wrote:
> On Tue, Dec 15, 2020 at 3:56 AM SeongJae Park <[email protected]> wrote:
> >
> > From: SeongJae Park <[email protected]>
> >
> > To avoid the unbounded increase of the overhead, DAMON groups adjacent
> > pages that assumed to have the same access frequencies into a region.
>
> 'that are assumed'
Good eye!
>
> > As long as the assumption (pages in a region have the same access
> > frequencies) is kept, only one page in the region is required to be
> > checked. Thus, for each ``sampling interval``,
> >
> > 1. the 'prepare_access_checks' primitive picks one page in each region,
> > 2. waits for one ``sampling interval``,
> > 3. checks whether the page is accessed meanwhile, and
> > 4. increases the access frequency of the region if so.
>
> I think you meant increasing the access 'count' or something.
> Increasing the access frequency somewhat conveys that the sampling
> interval is being decreased.
You're right, I will reword this in the next version.
>
> >
> > Therefore, the monitoring overhead is controllable by adjusting the
> > number of regions. DAMON allows both the underlying primitives and user
> > callbacks adjust regions for the trade-off. In other words, this commit
>
> 'callbacks to adjust'
Nice catch!
>
>
> > makes DAMON to use not only time-based sampling but also space-based
> > sampling.
> >
> > This scheme, however, cannot preserve the quality of the output if the
> > assumption is not guaranteed. Next commit will address this problem.
> >
> > Another problem of this region abstraction is additional memory space
> > overhead for the regions metadata. For example, suppose page
> > granularity monitoring that doesn't want to know fine-grained access
> > frequency but only if each page accessed or not.
>
> You mean when the sampling interval is equal to the aggregation interval, right?
Right, that would be a straightforward way to get the information with region
abstraction.
>
> > Then, we can do that
> > by directly resetting and reading the PG_Idle flags and/or the PTE
> > Accessed bits. The metadata for the region abstraction is only burden
> > in the case. For the reason, this commit makes DAMON to support the
> > user-defined arbitrary target, which could be stored in a void pointer
> > of the monitoring context with specific target type.
>
> Sorry I didn't follow. How does sampling interval equal to aggregation
> interval require user-defined arbitrary targets?
So, setting sampling interval equal to aggregation interval is a
straightforward way to get the if-accessed-only information with the
region-based sampling. However, this will waste memory with region metadata
(observed accesses counter). The wastage becomes much worse if we do
page-granularity monitoring, because the region metadata for start address and
end address of the regions would not necessary.
Someone can implement and use monitoring primitives making no such waste by
using thir own abstraction rather than the regions abstraction (and therefore
no regions metadata). To allow that, we make the regions abstraction to be
used only when the context is configured for special target type
(DAMON_REGION_SAMPLING_TARGET) and allow users to use their own arbitrary
abstraction with the arbitrary target type.
An RFC patchset for an example implementation of the arbitrary target type is
available:
https://lore.kernel.org/linux-mm/[email protected]/
>
> >
> > Signed-off-by: SeongJae Park <[email protected]>
> > Reviewed-by: Leonard Foerster <[email protected]>
> > ---
> > include/linux/damon.h | 109 ++++++++++++++++++++++++++++++--
> > mm/damon/core.c | 142 +++++++++++++++++++++++++++++++++++++++++-
> > 2 files changed, 243 insertions(+), 8 deletions(-)
> >
> > diff --git a/include/linux/damon.h b/include/linux/damon.h
> > index 387fa4399fc8..7d4685adc8a9 100644
> > --- a/include/linux/damon.h
> > +++ b/include/linux/damon.h
> > @@ -12,6 +12,48 @@
> > #include <linux/time64.h>
> > #include <linux/types.h>
> >
> > +/**
> > + * struct damon_addr_range - Represents an address region of [@start, @end).
> > + * @start: Start address of the region (inclusive).
> > + * @end: End address of the region (exclusive).
> > + */
> > +struct damon_addr_range {
> > + unsigned long start;
> > + unsigned long end;
> > +};
> > +
> > +/**
> > + * struct damon_region - Represents a monitoring target region.
> > + * @ar: The address range of the region.
> > + * @sampling_addr: Address of the sample for the next access check.
> > + * @nr_accesses: Access frequency of this region.
> > + * @list: List head for siblings.
> > + */
> > +struct damon_region {
> > + struct damon_addr_range ar;
> > + unsigned long sampling_addr;
> > + unsigned int nr_accesses;
> > + struct list_head list;
> > +};
> > +
> > +/**
> > + * struct damon_target - Represents a monitoring target.
> > + * @id: Unique identifier for this target.
> > + * @regions_list: Head of the monitoring target regions of this target.
> > + * @list: List head for siblings.
> > + *
> > + * Each monitoring context could have multiple targets. For example, a context
> > + * for virtual memory address spaces could have multiple target processes. The
> > + * @id of each target should be unique among the targets of the context. For
> > + * example, in the virtual address monitoring context, it could be a pidfd or
> > + * an address of an mm_struct.
> > + */
> > +struct damon_target {
> > + unsigned long id;
> > + struct list_head regions_list;
> > + struct list_head list;
> > +};
> > +
> > struct damon_ctx;
> >
> > /**
> > @@ -36,7 +78,8 @@ struct damon_ctx;
> > *
> > * @init_target_regions should construct proper monitoring target regions and
> > * link those to the DAMON context struct. The regions should be defined by
> > - * user and saved in @damon_ctx.target.
> > + * user and saved in @damon_ctx.arbitrary_target if @damon_ctx.target_type is
> > + * &DAMON_ARBITRARY_TARGET. Otherwise, &struct damon_region should be used.
> > * @update_target_regions should update the monitoring target regions for
> > * current status.
> > * @prepare_access_checks should manipulate the monitoring regions to be
> > @@ -46,7 +89,8 @@ struct damon_ctx;
> > * @reset_aggregated should reset the access monitoring results that aggregated
> > * by @check_accesses.
> > * @target_valid should check whether the target is still valid for the
> > - * monitoring.
> > + * monitoring. It receives &damon_ctx.arbitrary_target or &struct damon_target
> > + * pointer depends on &damon_ctx.target_type.
> > * @cleanup is called from @kdamond just before its termination. After this
> > * call, only @kdamond_lock and @kdamond will be touched.
> > */
> > @@ -91,6 +135,17 @@ struct damon_callback {
> > int (*before_terminate)(struct damon_ctx *context);
> > };
> >
> > +/**
> > + * enum damon_target_type - Represents the type of the monitoring target.
> > + *
> > + * @DAMON_REGION_SAMPLING_TARGET: Region based sampling target.
> > + * @DAMON_ARBITRARY_TARGET: User-defined arbitrary type target.
> > + */
> > +enum damon_target_type {
> > + DAMON_REGION_SAMPLING_TARGET,
> > + DAMON_ARBITRARY_TARGET,
>
> I would suggest removing the arbitrary target in this pathset. This
> patchset is only adding the region sampling target, so no need to add
> the arbitrary target here.
I think arbitrary targt type is necessary for above mentioned case. Also, this
makes backward compatible for the previous patch. However, as this patchset
doesn't introduce the real use of the arbitrary target type, I think it's also
ok to introduce this later. So I will drop this as you suggested in the next
version.
[...]
> > +
> > +/*
> > + * Add a region between two other regions
> > + */
> > +inline void damon_insert_region(struct damon_region *r,
> > + struct damon_region *prev, struct damon_region *next)
> > +{
> > + __list_add(&r->list, &prev->list, &next->list);
> > +}
> > +
> > +void damon_add_region(struct damon_region *r, struct damon_target *t)
> > +{
> > + list_add_tail(&r->list, &t->regions_list);
> > +}
> > +
>
> I don't see the benefit of these one line functions at least the following two.
We might want to use different data structures such as rbtree for regions
later. So I want to make the programming interface independent of the data
structure. This wrappers would help that.
[...]
Thanks,
SeongJae Park
On Mon, 1 Feb 2021 09:37:33 -0800 Shakeel Butt <[email protected]> wrote:
> On Tue, Dec 15, 2020 at 3:57 AM SeongJae Park <[email protected]> wrote:
> >
> > From: SeongJae Park <[email protected]>
> >
> > Even somehow the initial monitoring target regions are well constructed
> > to fulfill the assumption (pages in same region have similar access
> > frequencies), the data access pattern can be dynamically changed. This
> > will result in low monitoring quality. To keep the assumption as much
> > as possible, DAMON adaptively merges and splits each region based on
> > their access frequency.
> >
> > For each ``aggregation interval``, it compares the access frequencies of
> > adjacent regions and merges those if the frequency difference is small.
> > Then, after it reports and clears the aggregated access frequency of
> > each region, it splits each region into two or three regions if the
> > total number of regions will not exceed the user-specified maximum
> > number of regions after the split.
>
> Should there be any concerns regarding the number of regions
> oscillating even when the access pattern of the application is not
> changing? Does the system converge to equilibrium state or does it not
> matter?
DAMON will continue splitting regions, but all the changes will be reverted by
merging. Because callbacks are called after merging finished, this would not
matter.
>
> >
> > In this way, DAMON provides its best-effort quality and minimal overhead
> > while keeping the upper-bound overhead that users set.
> >
> > Signed-off-by: SeongJae Park <[email protected]>
> > Reviewed-by: Leonard Foerster <[email protected]>
> > ---
> > include/linux/damon.h | 41 +++++---
> > mm/damon/core.c | 220 ++++++++++++++++++++++++++++++++++++++++--
> > 2 files changed, 240 insertions(+), 21 deletions(-)
> >
> > diff --git a/include/linux/damon.h b/include/linux/damon.h
> > index 7d4685adc8a9..f446f8433599 100644
> > --- a/include/linux/damon.h
> > +++ b/include/linux/damon.h
> > @@ -12,6 +12,9 @@
> > #include <linux/time64.h>
> > #include <linux/types.h>
> >
> > +/* Minimal region size. Every damon_region is aligned by this. */
> > +#define DAMON_MIN_REGION PAGE_SIZE
> > +
> > /**
> > * struct damon_addr_range - Represents an address region of [@start, @end).
> > * @start: Start address of the region (inclusive).
> > @@ -86,6 +89,8 @@ struct damon_ctx;
> > * prepared for the next access check.
> > * @check_accesses should check the accesses to each region that made after the
> > * last preparation and update the number of observed accesses of each region.
> > + * It should also return max number of observed accesses that made as a result
> > + * of its update.
>
> Why?
To get the max access count without additional iteration of regions. The count
will be used to calculate merge/split threshold. I will add this explanation
in the next version.
The additional iteration would not be a real performance bottleneck for usual
case, so I we could make this optimization later. However, because making such
optimization with callback interface would cause some backward compatibility
issue, I'd like to do this now.
>
> > * @reset_aggregated should reset the access monitoring results that aggregated
> > * by @check_accesses.
> > * @target_valid should check whether the target is still valid for the
[...]
> >
> > +unsigned int damon_nr_regions(struct damon_target *t)
> > +{
> > + struct damon_region *r;
> > + unsigned int nr_regions = 0;
> > +
> > + damon_for_each_region(r, t)
> > + nr_regions++;
>
> Why not just add the region_count filed in damon_target?
Just to make the code simpler. We can easily optimize in the way if this turns
out to be a real performance bottleneck.
>
> > +
> > + return nr_regions;
> > +}
> > +
> > struct damon_ctx *damon_new_ctx(enum damon_target_type type)
> > {
> > struct damon_ctx *ctx;
> > @@ -128,8 +143,12 @@ struct damon_ctx *damon_new_ctx(enum damon_target_type type)
> > mutex_init(&ctx->kdamond_lock);
> >
> > ctx->target_type = type;
> > - if (type != DAMON_ARBITRARY_TARGET)
> > - INIT_LIST_HEAD(&ctx->region_targets);
> > + if (type != DAMON_ARBITRARY_TARGET) {
> > + ctx->min_nr_regions = 10;
> > + ctx->max_nr_regions = 1000;
>
> IMO these settings/heuristics should be part of the virtual address
> space monitor primitives and not be in the core monitor.
These are just default values. For the adpative regions adjustment, I think we
agreed on adding it in the core for now.
>
> > +
> > + INIT_LIST_HEAD(&ctx->adaptive_targets);
> > + }
> >
> > return ctx;
> > }
[...]
> > +
> > +/*
> > + * Split every target region into randomly-sized small regions
> > + *
> > + * This function splits every target region into random-sized small regions if
> > + * current total number of the regions is equal or smaller than half of the
> > + * user-specified maximum number of regions. This is for maximizing the
> > + * monitoring accuracy under the dynamically changeable access patterns. If a
> > + * split was unnecessarily made, later 'kdamond_merge_regions()' will revert
> > + * it.
> > + */
> > +static void kdamond_split_regions(struct damon_ctx *ctx)
> > +{
> > + struct damon_target *t;
> > + unsigned int nr_regions = 0;
> > + static unsigned int last_nr_regions;
> > + int nr_subregions = 2;
> > +
> > + damon_for_each_target(t, ctx)
> > + nr_regions += damon_nr_regions(t);
> > +
> > + if (nr_regions > ctx->max_nr_regions / 2)
> > + return;
>
> Shouldn't the limits on region be per-target instead of for the whole context?
I think this makes the monitoring overhead upperbound setting simpler. If we
need to set per-target monitoring upperbound, we can use multiple contexts for
each target.
>
> > +
> > + /* Maybe the middle of the region has different access frequency */
> > + if (last_nr_regions == nr_regions &&
> > + nr_regions < ctx->max_nr_regions / 3)
> > + nr_subregions = 3;
> > +
> > + damon_for_each_target(t, ctx)
> > + damon_split_regions_of(ctx, t, nr_subregions);
> > +
> > + last_nr_regions = nr_regions;
> > +}
> > +
> > /*
> > * Check whether it is time to check and apply the target monitoring regions
> > *
> > @@ -391,6 +588,8 @@ static int kdamond_fn(void *data)
> > struct damon_ctx *ctx = (struct damon_ctx *)data;
> > struct damon_target *t;
> > struct damon_region *r, *next;
> > + unsigned int max_nr_accesses = 0;
> > + unsigned long sz_limit = 0;
> >
> > pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
> >
> > @@ -399,6 +598,8 @@ static int kdamond_fn(void *data)
> > if (ctx->callback.before_start && ctx->callback.before_start(ctx))
> > set_kdamond_stop(ctx);
> >
> > + sz_limit = damon_region_sz_limit(ctx);
> > +
> > while (!kdamond_need_stop(ctx)) {
> > if (ctx->primitive.prepare_access_checks)
> > ctx->primitive.prepare_access_checks(ctx);
> > @@ -409,14 +610,20 @@ static int kdamond_fn(void *data)
> > usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
> >
> > if (ctx->primitive.check_accesses)
> > - ctx->primitive.check_accesses(ctx);
> > + max_nr_accesses = ctx->primitive.check_accesses(ctx);
> >
> > if (kdamond_aggregate_interval_passed(ctx)) {
> > + if (ctx->target_type != DAMON_ARBITRARY_TARGET)
> > + kdamond_merge_regions(ctx,
> > + max_nr_accesses / 10,
>
> What's the reason behind this 10?
It came from my gut feeling and it is still there because it worked well with
my test workloads. I think we could change that or allow users adjustable if
problematic case is found later.
Thanks,
SeongJae Park
[...]
On Mon, 1 Feb 2021 09:37:39 -0800 Shakeel Butt <[email protected]> wrote:
> On Tue, Dec 15, 2020 at 3:59 AM SeongJae Park <[email protected]> wrote:
> >
> > From: SeongJae Park <[email protected]>
> >
> > DAMON is designed to be used by kernel space code such as the memory
> > management subsystems, and therefore it provides only kernel space API.
>
> Which kernel space APIs are being referred here?
The symbols in 'include/linux/damon.h'
>
> > That said, letting the user space control DAMON could provide some
> > benefits to them. For example, it will allow user space to analyze
> > their specific workloads and make their own special optimizations.
> >
> > For such cases, this commit implements a simple DAMON application kernel
> > module, namely 'damon-dbgfs', which merely wraps the DAMON api and
> > exports those to the user space via the debugfs.
> >
[...]
> >
> > Signed-off-by: SeongJae Park <[email protected]>
> > Reviewed-by: Leonard Foerster <[email protected]>
> > ---
> > include/linux/damon.h | 3 +
> > mm/damon/Kconfig | 9 ++
> > mm/damon/Makefile | 1 +
> > mm/damon/core.c | 45 ++++++
> > mm/damon/dbgfs.c | 366 ++++++++++++++++++++++++++++++++++++++++++
> > 5 files changed, 424 insertions(+)
> > create mode 100644 mm/damon/dbgfs.c
> >
> > diff --git a/include/linux/damon.h b/include/linux/damon.h
> > index 39b4d6d3ddee..f9e0d4349352 100644
> > --- a/include/linux/damon.h
> > +++ b/include/linux/damon.h
> > @@ -265,9 +265,12 @@ unsigned int damon_nr_regions(struct damon_target *t);
> >
> > struct damon_ctx *damon_new_ctx(enum damon_target_type type);
> > void damon_destroy_ctx(struct damon_ctx *ctx);
> > +int damon_set_targets(struct damon_ctx *ctx,
> > + unsigned long *ids, ssize_t nr_ids);
> > int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> > unsigned long aggr_int, unsigned long regions_update_int,
> > unsigned long min_nr_reg, unsigned long max_nr_reg);
> > +int damon_nr_running_ctxs(void);
> >
> > int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
> > int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
> > diff --git a/mm/damon/Kconfig b/mm/damon/Kconfig
> > index 8ae080c52950..72f1683ba0ee 100644
> > --- a/mm/damon/Kconfig
> > +++ b/mm/damon/Kconfig
> > @@ -21,4 +21,13 @@ config DAMON_VADDR
> > This builds the default data access monitoring primitives for DAMON
> > that works for virtual address spaces.
> >
> > +config DAMON_DBGFS
> > + bool "DAMON debugfs interface"
> > + depends on DAMON_VADDR && DEBUG_FS
> > + help
> > + This builds the debugfs interface for DAMON. The user space admins
> > + can use the interface for arbitrary data access monitoring.
> > +
> > + If unsure, say N.
> > +
> > endmenu
> > diff --git a/mm/damon/Makefile b/mm/damon/Makefile
> > index 6ebbd08aed67..fed4be3bace3 100644
> > --- a/mm/damon/Makefile
> > +++ b/mm/damon/Makefile
> > @@ -2,3 +2,4 @@
> >
> > obj-$(CONFIG_DAMON) := core.o
> > obj-$(CONFIG_DAMON_VADDR) += vaddr.o
> > +obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
> > diff --git a/mm/damon/core.c b/mm/damon/core.c
> > index 5ca9f79ccbb6..b9575a6bebff 100644
> > --- a/mm/damon/core.c
> > +++ b/mm/damon/core.c
> > @@ -166,6 +166,37 @@ void damon_destroy_ctx(struct damon_ctx *ctx)
> > kfree(ctx);
> > }
> >
> > +/**
> > + * damon_set_targets() - Set monitoring targets.
> > + * @ctx: monitoring context
> > + * @ids: array of target ids
> > + * @nr_ids: number of entries in @ids
> > + *
> > + * This function should not be called while the kdamond is running.
> > + *
> > + * Return: 0 on success, negative error code otherwise.
> > + */
> > +int damon_set_targets(struct damon_ctx *ctx,
> > + unsigned long *ids, ssize_t nr_ids)
> > +{
> > + ssize_t i;
> > + struct damon_target *t, *next;
> > +
> > + damon_for_each_target_safe(t, next, ctx)
> > + damon_destroy_target(t);
>
> You need to put the reference on the target before destroying.
Oops, you're right. I will fix this in the next version.
>
> > +
> > + for (i = 0; i < nr_ids; i++) {
> > + t = damon_new_target(ids[i]);
> > + if (!t) {
> > + pr_err("Failed to alloc damon_target\n");
> > + return -ENOMEM;
> > + }
> > + damon_add_target(ctx, t);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > /**
> > * damon_set_attrs() - Set attributes for the monitoring.
> > * @ctx: monitoring context
> > @@ -206,6 +237,20 @@ int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
> > return 0;
> > }
> >
> > +/**
> > + * damon_nr_running_ctxs() - Return number of currently running contexts.
> > + */
> > +int damon_nr_running_ctxs(void)
> > +{
> > + int nr_ctxs;
> > +
> > + mutex_lock(&damon_lock);
> > + nr_ctxs = nr_running_ctxs;
> > + mutex_unlock(&damon_lock);
>
> READ_ONCE(nr_running_ctxs) ?
I'd like to keep the code simpler to read, unless this turns out to be a real
performance bottleneck.
>
> > +
> > + return nr_ctxs;
> > +}
> > +
[...]
> > +
> > +static ssize_t dbgfs_target_ids_write(struct file *file,
> > + const char __user *buf, size_t count, loff_t *ppos)
> > +{
> > + struct damon_ctx *ctx = file->private_data;
> > + char *kbuf, *nrs;
> > + unsigned long *targets;
> > + ssize_t nr_targets;
> > + ssize_t ret = count;
> > + int i;
> > + int err;
> > +
> > + kbuf = user_input_str(buf, count, ppos);
> > + if (IS_ERR(kbuf))
> > + return PTR_ERR(kbuf);
> > +
> > + nrs = kbuf;
> > +
> > + targets = str_to_target_ids(nrs, ret, &nr_targets);
> > + if (!targets) {
> > + ret = -ENOMEM;
> > + goto out;
> > + }
> > +
> > + if (targetid_is_pid(ctx)) {
> > + for (i = 0; i < nr_targets; i++)
> > + targets[i] = (unsigned long)find_get_pid(
> > + (int)targets[i]);
> > + }
> > +
> > + mutex_lock(&ctx->kdamond_lock);
> > + if (ctx->kdamond) {
> > + ret = -EINVAL;
> > + goto unlock_out;
>
> You need to put_pid on the targets array.
Good catch!
>
> > + }
> > +
> > + err = damon_set_targets(ctx, targets, nr_targets);
> > + if (err)
> > + ret = err;
>
> You need to handle the partial failure from damon_set_targets().
My intention is to keep partial success as is.
>
>
> > +unlock_out:
> > + mutex_unlock(&ctx->kdamond_lock);
> > + kfree(targets);
> > +out:
> > + kfree(kbuf);
> > + return ret;
> > +}
[...]
Thanks,
SeongJae Park
On Tue, 15 Dec 2020 12:54:43 +0100 SeongJae Park <[email protected]> wrote:
> From: SeongJae Park <[email protected]>
>
> In some use cases, users would want to run multiple monitoring context.
> For example, if a user wants a high precision monitoring and dedicating
> multiple CPUs for the job is ok, because DAMON creates one monitoring
> thread per one context, the user can split the monitoring target regions
> into multiple small regions and create one context for each region. Or,
> someone might want to simultaneously monitor different address spaces,
> e.g., both virtual address space and physical address space.
>
> The DAMON's API allows such usage, but 'damon-dbgfs' does not.
> Therefore, only kernel space DAMON users can do multiple contexts
> monitoring.
>
> This commit allows the user space DAMON users to use multiple contexts
> monitoring by introducing two new 'damon-dbgfs' debugfs files,
> 'mk_context' and 'rm_context'. Users can create a new monitoring
> context by writing the desired name of the new context to 'mk_context'.
> Then, a new directory with the name and having the files for setting of
> the context ('attrs', 'target_ids' and 'record') will be created under
> the debugfs directory. Writing the name of the context to remove to
> 'rm_context' will remove the related context and directory.
>
> Signed-off-by: SeongJae Park <[email protected]>
> ---
> mm/damon/dbgfs.c | 213 ++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 211 insertions(+), 2 deletions(-)
>
> diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c
> index a1512d3e5afe..46186b057e1a 100644
> --- a/mm/damon/dbgfs.c
> +++ b/mm/damon/dbgfs.c
[...]
> +/*
> + * Remove a context of @name and its debugfs directory.
> + *
> + * This function should be called while holding damon_dbgfs_lock.
> + *
> + * Return 0 on success, negative error code otherwise.
> + */
> +static int dbgfs_rm_context(char *name)
> +{
> + struct dentry *root, *dir, **new_dirs;
> + struct damon_ctx **new_ctxs;
> + int i, j;
> +
> + if (damon_nr_running_ctxs())
> + return -EBUSY;
> +
> + root = dbgfs_dirs[0];
> + if (!root)
> + return -ENOENT;
> +
> + dir = debugfs_lookup(name, root);
> + if (!dir)
> + return -ENOENT;
> +
> + new_dirs = kmalloc_array(dbgfs_nr_ctxs - 1, sizeof(*dbgfs_dirs),
> + GFP_KERNEL);
> + if (!new_dirs)
> + return -ENOMEM;
> +
> + new_ctxs = kmalloc_array(dbgfs_nr_ctxs - 1, sizeof(*dbgfs_ctxs),
> + GFP_KERNEL);
> + if (!new_ctxs) {
> + kfree(new_dirs);
> + return -ENOMEM;
> + }
> +
> + for (i = 0, j = 0; i < dbgfs_nr_ctxs; i++) {
> + if (dbgfs_dirs[i] == dir) {
> + debugfs_remove(dbgfs_dirs[i]);
> + dbgfs_destroy_ctx(dbgfs_ctxs[i]);
In case of virtual address monitoring, dbgfs_destroy_ctx() calls
damon_destroy_ctx() and damon_destroy_ctx() doesn't put the target pids. I
will fix this in the next version.
> + continue;
> + }
> + new_dirs[j] = dbgfs_dirs[i];
> + new_ctxs[j++] = dbgfs_ctxs[i];
> + }
> +
> + kfree(dbgfs_dirs);
> + kfree(dbgfs_ctxs);
> +
> + dbgfs_dirs = new_dirs;
> + dbgfs_ctxs = new_ctxs;
> + dbgfs_nr_ctxs--;
> +
> + return 0;
> +}
Thanks,
SeongJae Park
[...]
On Tue, 2 Feb 2021 07:07:24 -0800 Shakeel Butt <[email protected]> wrote:
> On Tue, Feb 2, 2021 at 2:30 AM SeongJae Park <[email protected]> wrote:
> >
> >> On Mon, 1 Feb 2021 09:37:39 -0800 Shakeel Butt <[email protected]> wrote:
> >>
> >>> On Tue, Dec 15, 2020 at 3:59 AM SeongJae Park <[email protected]> wrote:
> >>>>
> >>>> From: SeongJae Park <[email protected]>
> >>>>
> >>>> DAMON is designed to be used by kernel space code such as the memory
> >>>> management subsystems, and therefore it provides only kernel space API.
> >>>
> >>> Which kernel space APIs are being referred here?
> >>
> >> The symbols in 'include/linux/damon.h'
> >>
> >>>
> >>>> That said, letting the user space control DAMON could provide some
> >>>> benefits to them. For example, it will allow user space to analyze
> >>>> their specific workloads and make their own special optimizations.
> >>>>
> >>>> For such cases, this commit implements a simple DAMON application kernel
> >>>> module, namely 'damon-dbgfs', which merely wraps the DAMON api and
> >>>> exports those to the user space via the debugfs.
> >>>>
> >> [...]
> >>>> +static ssize_t dbgfs_target_ids_write(struct file *file,
> >>>> + const char __user *buf, size_t count, loff_t *ppos)
> >>>> +{
> >>>> + struct damon_ctx *ctx = file->private_data;
> >>>> + char *kbuf, *nrs;
> >>>> + unsigned long *targets;
> >>>> + ssize_t nr_targets;
> >>>> + ssize_t ret = count;
> >>>> + int i;
> >>>> + int err;
> >>>> +
> >>>> + kbuf = user_input_str(buf, count, ppos);
> >>>> + if (IS_ERR(kbuf))
> >>>> + return PTR_ERR(kbuf);
> >>>> +
> >>>> + nrs = kbuf;
> >>>> +
> >>>> + targets = str_to_target_ids(nrs, ret, &nr_targets);
> >>>> + if (!targets) {
> >>>> + ret = -ENOMEM;
> >>>> + goto out;
> >>>> + }
> >>>> +
> >>>> + if (targetid_is_pid(ctx)) {
> >>>> + for (i = 0; i < nr_targets; i++)
> >>>> + targets[i] = (unsigned long)find_get_pid(
> >>>> + (int)targets[i]);
> >>>> + }
> >>>> +
> >>>> + mutex_lock(&ctx->kdamond_lock);
> >>>> + if (ctx->kdamond) {
> >>>> + ret = -EINVAL;
> >>>> + goto unlock_out;
> >>>
> >>> You need to put_pid on the targets array.
> >>
> >> Good catch!
> >>
> >>>
> >>>> + }
> >>>> +
> >>>> + err = damon_set_targets(ctx, targets, nr_targets);
> >>>> + if (err)
> >>>> + ret = err;
> >>>
> >>> You need to handle the partial failure from damon_set_targets().
> >>
> >> My intention is to keep partial success as is.
> >
> > But, we should put_pid() partial failures... I will simply make this to
> > completely fail with no registered target.
> >
>
> You can simplify by simply restricting to one pid/target per each write syscall.
Right, thanks for the suggestion. However, I already almost finished writing
the fix. If there is no other concern, I'd like to keep current interface.
Thanks,
SeongJae Park
On Tue, Feb 2, 2021 at 7:46 AM SeongJae Park <[email protected]> wrote:
>
[snip]
> >
> > You can simplify by simply restricting to one pid/target per each write syscall.
>
> Right, thanks for the suggestion. However, I already almost finished writing
> the fix. If there is no other concern, I'd like to keep current interface.
>
>
Please go ahead with the current.
