From:   "Huang, Ying" <ying.huang@intel.com>
To:     Bharata B Rao <bharata@amd.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
        <mgorman@suse.de>, <mingo@redhat.com>, <bp@alien8.de>,
        <dave.hansen@linux.intel.com>, <x86@kernel.org>,
        <akpm@linux-foundation.org>, <luto@kernel.org>,
        <tglx@linutronix.de>, <yue.li@memverge.com>,
        <Ravikumar.Bangoria@amd.com>
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
References: <20230208073533.715-1-bharata@amd.com>
        <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
        <a0127c3c-22c9-ac46-1e9f-200978fec392@amd.com>
Date:   Mon, 13 Feb 2023 10:56:17 +0800
In-Reply-To: <a0127c3c-22c9-ac46-1e9f-200978fec392@amd.com> (Bharata B. Rao's
        message of "Thu, 9 Feb 2023 11:27:40 +0530")
Message-ID: <87cz6eb7e6.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii
Precedence: bulk

Bharata B Rao <bharata@amd.com> writes:

> On 2/8/2023 11:33 PM, Peter Zijlstra wrote:
>> On Wed, Feb 08, 2023 at 01:05:28PM +0530, Bharata B Rao wrote:
>> 
>> 
>>> - Hardware provided access information could be very useful for driving
>>>   hot page promotion in tiered memory systems. Need to check if this
>>>   requires different tuning/heuristics apart from what NUMA balancing
>>>   already does.
>> 
>> I think Huang Ying looked at that from the Intel POV and I think the
>> conclusion was that it doesn't really work out. What you need is
>> frequency information, but the PMU doesn't really give you that. You
>> need to process a *ton* of PMU data in-kernel.
>
> What I am doing here is to feed the access data into NUMA balancing which
> already has the logic to aggregate that at task and numa group level and
> decide if that access is actionable in terms of migrating the page. In this
> context, I am not sure about the frequency information that you and Dave
> are mentioning. AFAIU, existing NUMA balancing takes care of taking
> action, IBS becomes an alternative source of access information to NUMA
> hint faults.

We do need frequency information to determine whether a page is hot
enough to be migrated to the fast memory (promotion).  What PMU provided
is just "recently" accessed pages, not "frequently" accessed pages.  For
current NUMA balancing implementation, please check
NUMA_BALANCING_MEMORY_TIERING in should_numa_migrate_memory().  In
general, it estimates the page access frequency via measuring the
latency between page table scanning and page fault, the shorter the
latency, the higher the frequency.  This isn't perfect, but provides a
starting point.  You need to consider how to get frequency information
via PMU.  For example, you may count access number for each page, aging
them periodically, and get hot threshold via some statistics.

Best Regards,
Huang, Ying