Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   SeongJae Park <sj38.park@gmail.com>
To:     Mel Gorman <mgorman@suse.de>
Cc:     SeongJae Park <sjpark@amazon.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Peter Zijlstra <peterz@infradead.org>,
        "Huang, Ying" <ying.huang@intel.com>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
        Rik van Riel <riel@surriel.com>,
        Daniel Jordan <daniel.m.jordan@oracle.com>,
        Tejun Heo <tj@kernel.org>, Dave Hansen <dave.hansen@intel.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Aubrey Li <aubrey.li@intel.com>
Subject: Re: Re: Re: Re: Re: [RFC] autonuma: Support to scan page table asynchronously
Date:   Sat, 18 Apr 2020 11:48:11 +0200
Message-Id: <20200418094811.4757-1-sj38.park@gmail.com>
In-Reply-To: <20200417144626.GA3711@suse.de> (raw)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, 17 Apr 2020 15:46:26 +0100 Mel Gorman <mgorman@suse.de> wrote:

> On Fri, Apr 17, 2020 at 02:44:37PM +0200, SeongJae Park wrote:
> > On Fri, 17 Apr 2020 13:16:29 +0100 Mel Gorman <mgorman@techsingularity.net> wrote:
> > 
> > > On Fri, Apr 17, 2020 at 12:21:29PM +0200, SeongJae Park wrote:
> > > > On Fri, 17 Apr 2020 12:04:17 +0200 Peter Zijlstra <peterz@infradead.org> wrote:
> > > > 
> > > > > On Fri, Apr 17, 2020 at 09:05:08AM +0200, SeongJae Park wrote:
> > > > > > I think the main idea of DAMON[1] might be able to applied here.  Have you
> > > > > > considered it?
> > > > > > 
> > > > > > [1] https://lore.kernel.org/linux-mm/20200406130938.14066-1-sjpark@amazon.com/
> > > > > 
> > > > > I've ignored that entire thing after you said the information it
> > > > > provides was already available through the PMU.
> > > > 
> > > > Sorry if my answer made you confused.  What I wanted to say was that the
> > > > fundamental access checking mechanism that DAMON depends on is PTE Accessed bit
> > > > for now, but it could be modified to use PMU or other features instead. 
> > > 
> > > I would not be inclined to lean towards either approach for NUMA
> > > balancing. Fiddling with the accessed bit can have consequences for page
> > > aging and residency -- fine for debugging a problem, not to fine for
> > > normal usage. I would expect the PMU approach would have high overhead
> > > as well as taking over a PMU counter that userspace debugging may expect
> > > to be available.
> > 
> > Good point.  But, isn't it ok to use Accessed bit as long as PG_Idle and
> > PG_Young is adjusted properly?  Current DAMON implementation does so, as
> > idle_page_tracking also does.
> > 
> 
> PG_idle and PG_young are page flags that are used by idle tracking. The
> accessed bit I'm thinking of is in the PTE bit and it's the PTE that
> is changed by page_referenced() during reclaim. So it's not clear how
> adjusting the page bits help avoid page aging problems during reclaim.

The idle tracking reads and updates not only PG_idle and PG_young, but also the
accessed bit.  So idle tacking was also required to deal with the interference
of the reclaimer logic and thus it introduced PG_young.  The commit
33c3fc71c8cf ("mm: introduce idle page tracking") explains this as below:

    The Young page flag is used to avoid interference with the memory
    reclaimer.  A page's Young flag is set whenever the Access bit of a page
    table entry pointing to the page is cleared by writing to the bitmap file.
    If page_referenced() is called on a Young page, it will add 1 to its
    return value, therefore concealing the fact that the Access bit was
    cleared.

DAMON adjusts PG_young in similar way to avoid the interference.  DAMON further
adjusts PG_idle so that it does not interfere to page idle tracking mechanism.
I think I made you confused by unnecessarily mentioning PG_idle, sorry.

> 
> Maybe your suggestion was to move NUMA balancing to use the PG_idle and
> PG_young bits from idle tracking but I'm not sure what that gains us.
> This may be because I did not look too closely at DAMON as for debugging
> and analysis, the PMU sounded more suitable.
> 
> It's not clear to me how looking at the page or pte bit handling of
> DAMON helps us reduce the scanning cost of numa balancing. There may
> be some benefit in splitting the address space and rescanning sections
> that were referenced but it has similar pitfalls to simply tracking the
> highest/lowest address that trapped a NUMA hinting fault.

DAMON allows users to know which address ranges have which access frequency.
Thus, I think DAMON could be used for detection of hot pages, which will be
good migration candidates, instead of the NUMA hinting fault based referenced
pages detection.

The benefits I expect from this change are better accuracy and less overhead.
As we can know not only referenced pages but hot pages, migration will be more
effective.

In terms of the overhead, DAMON allows users to set the upperbound of the
monitoring overhead regardless of the size of the targe address space, unlike
the page scanning based mechanisms.  Thus, the overhead could be reduced,
especially if the target process has large memory footprint.

As I'm not deeply understanding AutoNUMA, I might saying something unrealistic
or missing some important points.


Thanks,
SeongJae Park

> 
> -- 
> Mel Gorman
> SUSE Labs