Message-ID: <4808d3fa-bb68-d4c8-681f-0b2770d78041@intel.com>
Date:   Wed, 8 Feb 2023 10:12:45 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
Content-Language: en-US
To:     Peter Zijlstra <peterz@infradead.org>,
        Bharata B Rao <bharata@amd.com>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@suse.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        x86@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        tglx@linutronix.de, yue.li@memverge.com,
        Ravikumar.Bangoria@amd.com, ying.huang@intel.com
References: <20230208073533.715-1-bharata@amd.com>
 <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
From:   Dave Hansen <dave.hansen@intel.com>
In-Reply-To: <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2/8/23 10:03, Peter Zijlstra wrote:
>> - Hardware provided access information could be very useful for driving
>>   hot page promotion in tiered memory systems. Need to check if this
>>   requires different tuning/heuristics apart from what NUMA balancing
>>   already does.
> I think Huang Ying looked at that from the Intel POV and I think the
> conclusion was that it doesn't really work out. What you need is
> frequency information, but the PMU doesn't really give you that. You
> need to process a *ton* of PMU data in-kernel.

Yeah, there were two big problems.

First, IIRC, Intel PEBS at the time only gave guest virtual addresses in
the PEBS records.  They had to be translated back to host addresses to
be usable.  That was extra expensive.

Second, it *did* take a lot of processing to turn raw memory accesses
into actionable frequency data.  That meant that we started in a hole
performance-wise and had to make *REALLY* good decisions about page
migration to make up for it.

The performance data here don't look awful, but they don't seem to add
up to a clear win either.  I'm having a hard time imagining who would
turn this on and how widely it would get used in practice.