Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   SeongJae Park <sjpark@amazon.com>
To:     Shakeel Butt <shakeelb@google.com>
CC:     SeongJae Park <sjpark@amazon.com>,
        SeongJae Park <sjpark@amazon.de>,
        <Jonathan.Cameron@huawei.com>,
        Andrea Arcangeli <aarcange@redhat.com>, <acme@kernel.org>,
        <alexander.shishkin@linux.intel.com>, <amit@kernel.org>,
        <benh@kernel.crashing.org>, <brendan.d.gregg@gmail.com>,
        Brendan Higgins <brendanhiggins@google.com>,
        Qian Cai <cai@lca.pw>,
        Colin Ian King <colin.king@canonical.com>,
        Jonathan Corbet <corbet@lwn.net>,
        "David Hildenbrand" <david@redhat.com>, <dwmw@amazon.com>,
        "Du, Fan" <fan.du@intel.com>, <foersleo@amazon.de>,
        Greg Thelen <gthelen@google.com>,
        Ian Rogers <irogers@google.com>, <jolsa@redhat.com>,
        "Kirill A. Shutemov" <kirill@shutemov.name>,
        <mark.rutland@arm.com>, Mel Gorman <mgorman@suse.de>,
        Minchan Kim <minchan@kernel.org>,
        Ingo Molnar <mingo@redhat.com>, <namhyung@kernel.org>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        Randy Dunlap <rdunlap@infradead.org>,
        Rik van Riel <riel@surriel.com>,
        David Rientjes <rientjes@google.com>,
        Steven Rostedt <rostedt@goodmis.org>, <rppt@kernel.org>,
        <sblbir@amazon.com>, <shuah@kernel.org>, <sj38.park@gmail.com>,
        <snu@amazon.de>, Vlastimil Babka <vbabka@suse.cz>,
        Vladimir Davydov <vdavydov.dev@gmail.com>,
        Yang Shi <yang.shi@linux.alibaba.com>,
        Huang Ying <ying.huang@intel.com>, <zgf574564920@gmail.com>,
        <linux-damon@amazon.com>, Linux MM <linux-mm@kvack.org>,
        <linux-doc@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>
Subject: Re: [RFC v7 00/10] DAMON: Support Physical Memory Address Space Monitoring
Date:   Thu, 20 Aug 2020 09:10:52 +0200
Message-ID: <20200820071052.24271-1-sjpark@amazon.com>
In-Reply-To: <CALvZod6RS66aSFjWHvpbjuinz2mwbGDnz+gh5L7dp+c3D_Zy1w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, 19 Aug 2020 18:21:44 -0700 Shakeel Butt <shakeelb@google.com> wrote:

> On Tue, Aug 18, 2020 at 12:25 AM SeongJae Park <sjpark@amazon.com> wrote:
> >
> > From: SeongJae Park <sjpark@amazon.de>
> >
> > Changes from Previous Version
> > =============================
> >
> > - Use 42 as the fake target id for paddr instead of -1
> > - Fix a typo
> >
> > Introduction
> > ============
> >
> > DAMON[1] programming interface users can extend DAMON for any address space by
> > configuring the address-space specific low level primitives with appropriate
> > ones including their own implementations.  However, because the implementation
> > for the virtual address space is only available now, the users should implement
> > their own for other address spaces.  Worse yet, the user space users who rely
> > on the debugfs interface and user space tool, cannot implement their own.
> >
> > This patchset implements another reference implementation of the low level
> > primitives for the physical memory address space.  With this change, hence, the
> > kernel space users can monitor both the virtual and the physical address spaces
> > by simply changing the configuration in the runtime.  Further, this patchset
> > links the implementation to the debugfs interface and the user space tool for
> > the user space users.
> >
> > Note that the implementation supports only the user memory, as same to the idle
> > page access tracking feature.
> >
> > [1] https://lore.kernel.org/linux-mm/20200706115322.29598-1-sjpark@amazon.com/
> >
> 
> I am still struggling to find the benefit of this feature the way it
> is implemented i.e. region based physical address space monitoring.
> What exactly am I supposed to do for a given hot (or cold) physical
> region? In a containerized world, that region can contain pages from
> any cgroup. I can not really do anything about the accesses PHY-DAMON
> provides me for a region.

Technically speaking, this patchset introduces an implementation of DAMON's low
level primitives for physical address space of LRU-listed pages.  In other
words, it is not designed for cgroups case.  Also, please note that this
patchset is only RFC, because it aims to only show the future plan of DAMON and
get opinions about the concept before being serious.  It will be serious only
after the DAMON patchset is merged.  Maybe I didn' made this point clear in the
CV, sorry.  I will state this clearly in the next spin.

However, owing to the flexible design of DAMON, you can still use DAMON for
cgroups case, though you need to make some efforts.  There could be a number of
ways.

First, you could figure out the physical address regions for the target
cgroups by yourself, set the target regions by yourself in 'damon_ctx' object
and pass it to 'damon_start()'.  For dynamic page allocations, you could check
if a monitored region belongs to your target cgroup or not from your action
making code, which could be implemented in the '->sample_cb()' or
'->aggregate_cb()' callbacks.

Note that you can even update the regions inside the callbacks.  That is, you
can remove regions containing pages of other containers, add new pages
allocated for your target containers, adjust regions having pages of both other
container and your target containers to represent only your target container's.

Second, you could expand DAMON for cgroups by implementing your own low level
primitives.  You could also reuse some of the current implementation.  For
example, you could implement only '->init_target_regions' and
'->update_target_regions' callbacks again so that only the pages of your target
cgroup belongs in the target regions.  However, if you need to monitor
non-LRU-listed pages, you should implement '->prepare_access_checks()' and
'->check_accesses()' callbacks.

> 
> Now if you give me per-page information that would be useful as I can
> at least get per-cgroup accesses (idle or re-use data) but that would
> be as costly as Page Idle Tracking.

So, seems you are saying about the 'adaptive regions adjustment' disabled page
granularity monitoring case.

Indeed.  Same information comes with same overhead.  Moreover, in the page
granularity monitoring case, DAMON will make more space overhead (at least 8
bytes per page), because DAMON will represent each page as a physical address
region having start address and end address, while Idle Pages Tracking can use
only pfn.  I'm planning optimizations for this page granularity case as a
future work.

However, if you don't strictly need page granularity accuracy, you could reduce
the overhead by using larger granularity.  That is, you can set the monitoring
granularity as you want while the adaptive regions adjustment is disabled.  You
could even use variable granularity in this case using the callbacks mentioned
above.


So, DAMON is a framework rather than a tool.  Though it comes with basic
applications using DAMON as a framework (e.g., the virtual address space low
primitives implementation, DAMON debugfs interface, and the DAMON user space
tool) that could be useful in simple use cases, you need to code your
application on it if your use cases are out of the simple cases.  I will also
develop more of such applications for more use-cases, but it will be only after
the framework is complete enough to be merged in the mainline.


Thanks,
SeongJae Park