From:   "Huang, Ying" <ying.huang@intel.com>
To:     Bharata B Rao <bharata@amd.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
        <mgorman@suse.de>, <mingo@redhat.com>, <bp@alien8.de>,
        <dave.hansen@linux.intel.com>, <x86@kernel.org>,
        <akpm@linux-foundation.org>, <luto@kernel.org>,
        <tglx@linutronix.de>, <yue.li@memverge.com>,
        <Ravikumar.Bangoria@amd.com>
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
References: <20230208073533.715-1-bharata@amd.com>
        <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
        <a0127c3c-22c9-ac46-1e9f-200978fec392@amd.com>
        <87cz6eb7e6.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <3c811078-c869-452a-8e2d-ebe720d21691@amd.com>
Date:   Mon, 13 Feb 2023 11:34:19 +0800
In-Reply-To: <3c811078-c869-452a-8e2d-ebe720d21691@amd.com> (Bharata B. Rao's
        message of "Mon, 13 Feb 2023 08:53:55 +0530")
Message-ID: <874jrqb5ms.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii
Precedence: bulk

Bharata B Rao <bharata@amd.com> writes:

> On 2/13/2023 8:26 AM, Huang, Ying wrote:
>> Bharata B Rao <bharata@amd.com> writes:
>> 
>>> On 2/8/2023 11:33 PM, Peter Zijlstra wrote:
>>>> On Wed, Feb 08, 2023 at 01:05:28PM +0530, Bharata B Rao wrote:
>>>>
>>>>
>>>>> - Hardware provided access information could be very useful for driving
>>>>>   hot page promotion in tiered memory systems. Need to check if this
>>>>>   requires different tuning/heuristics apart from what NUMA balancing
>>>>>   already does.
>>>>
>>>> I think Huang Ying looked at that from the Intel POV and I think the
>>>> conclusion was that it doesn't really work out. What you need is
>>>> frequency information, but the PMU doesn't really give you that. You
>>>> need to process a *ton* of PMU data in-kernel.
>>>
>>> What I am doing here is to feed the access data into NUMA balancing which
>>> already has the logic to aggregate that at task and numa group level and
>>> decide if that access is actionable in terms of migrating the page. In this
>>> context, I am not sure about the frequency information that you and Dave
>>> are mentioning. AFAIU, existing NUMA balancing takes care of taking
>>> action, IBS becomes an alternative source of access information to NUMA
>>> hint faults.
>> 
>> We do need frequency information to determine whether a page is hot
>> enough to be migrated to the fast memory (promotion).  What PMU provided
>> is just "recently" accessed pages, not "frequently" accessed pages.  For
>> current NUMA balancing implementation, please check
>> NUMA_BALANCING_MEMORY_TIERING in should_numa_migrate_memory().  In
>> general, it estimates the page access frequency via measuring the
>> latency between page table scanning and page fault, the shorter the
>> latency, the higher the frequency.  This isn't perfect, but provides a
>> starting point.  You need to consider how to get frequency information
>> via PMU.  For example, you may count access number for each page, aging
>> them periodically, and get hot threshold via some statistics.
>
> For the tiered memory hot page promotion case of NUMA balancing, we will
> have to maintain frequency information in software when such information
> isn't available from the hardware.

Yes.  It's challenging to calculate frequency information.  Please
consider how to do that.

Best Regards,
Huang, Ying