Message-ID: <3c811078-c869-452a-8e2d-ebe720d21691@amd.com>
Date:   Mon, 13 Feb 2023 08:53:55 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
Content-Language: en-US
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@suse.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        x86@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        tglx@linutronix.de, yue.li@memverge.com, Ravikumar.Bangoria@amd.com
References: <20230208073533.715-1-bharata@amd.com>
 <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
 <a0127c3c-22c9-ac46-1e9f-200978fec392@amd.com>
 <87cz6eb7e6.fsf@yhuang6-desk2.ccr.corp.intel.com>
From:   Bharata B Rao <bharata@amd.com>
In-Reply-To: <87cz6eb7e6.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QTVaUG9jQ0I4eGlxcEMrQUlWZWdEOGpaMk90RitudDBSZEllYWNHaXhLa3lQ?=
 =?utf-8?B?czNqZm0yTGg3dGhSTjZlSXp4a1RWenlDVlJnNG5SenVsSS9Bb1RSNFlhVXlQ?=
 =?utf-8?B?Yk1FaXNCeWtSRGZKV250UzFrblkxN1ptbzhLRzhqUjkxTXVIMVArdzdnNW5z?=
 =?utf-8?B?eUZ1UUFIbmlUbytHQWdVZXFGSUxvdTY4NTVFUml0b0UvY2tMUDJ3clNLMkhr?=
 =?utf-8?B?RzJwaWNSZDN0dWxEWnJLeDB0YzdCNUpyeW1abWVDNTF6ZE80bHJaNVVxakIz?=
 =?utf-8?B?elRINlBzU3BpMVNUQnIvWTYveDM1aEp0dFR1UWo4amtHdEIra1FKYjhaMFBX?=
 =?utf-8?B?MCtnQ0F6a3B4TzltRlRyOXF0MkpjV3pUUkVpRTRKRkhVcDg5N3Ivbm9idVAw?=
 =?utf-8?B?MU0yVitFQmlJMTEwM0NkMTk1WWF0YkVidnVId1Z6VWZFK2gvT1hCdk03UmtJ?=
 =?utf-8?B?eExpeUJpc0gxaU15djNyb1VscnBwUVl1bkNSc3NyMDkwUWRiVmt5b1FxNU9V?=
 =?utf-8?B?VDUyQVpLVHJiT0FrRm1QSUhzMTVlNTRDM20vU0xUMUtxakZDZG1xV3JRcyt6?=
 =?utf-8?B?T1hNN3hmaTF0VHZ3MmtxZUdDOFZQSHZmMHRLOFRFdFRPbjIyK2h5YW9TeTBr?=
 =?utf-8?B?aG8xTXpqeGg4cXN1L2xkT1QyNXgrNEp2RG1NZ1NzR0Jwdzc2MWxEa0wzdU0y?=
 =?utf-8?B?WW9lbk1lbk9TK0NTT29UeUZhMzJJbTJwV3VXN05PTnZldnp1SEVhbDJqYUR4?=
 =?utf-8?B?WjlRaEtVQUJ1VlI4anN2bTVBdE8rcEFZRnZYTDhEeDlTYkdrdXM5ekRTL3RR?=
 =?utf-8?B?b01ZT1pLdjJFNVUzZjkrbGFpK3BrY2ZnZGFGWXNFL0tzVFc4UEFjNnkreFNO?=
 =?utf-8?B?QUpjTHk3NHk4SmE1eCtkMWVJbkNaT0YxaExhTFVnRFBQT1FPUDNublBvblEw?=
 =?utf-8?B?NlBtVEppS3VwWkd4dTBLck5YYVBTdGtQT21BY28yTVdyWURoR2RtZ2FMMzBN?=
 =?utf-8?B?bHdRWkpDWWlEV08xOHVKbXh5RXNIR1lPeEtBVGlkTVVGNHVtQ2tUMGFyalph?=
 =?utf-8?B?NWpsMlE4SDRQdTZPNXRtMElpMnNodWpCalJiK3FSOGowYjhsWlFOUHVWV0FQ?=
 =?utf-8?B?N0xlVVY4VjlkYklVZ0paMUcxVXhUUHBzb1dXM3VqdW8vWW5mb0xrNjhpWjhJ?=
 =?utf-8?B?Y0lIam0wc1JndEdwUFBrY2w4dUE0SG9EYVd2ZTZmTWRGQlo5aDZpZWNKdXFL?=
 =?utf-8?B?dWZ1R3Z0TGVWWHZ0ZnlpYXlUeEFZN2VNY3plTEVHSC80TXJ6QjRCVi82Ynh6?=
 =?utf-8?B?bEYrZnRrRFpTUExENW11RHZXUWErTUNjMUdhVlB1OG1sTTZVZFZ1aGtzd2tY?=
 =?utf-8?B?SVRZMUUxUHpxdzFpM3pwa3lmNVd6K29ESjJ3akw5VDNVTUg5blAvZGduUFpO?=
 =?utf-8?B?N3Zydk5KL3VTM292eU93SW1CK3VsNVV2dnQ0VllMRDNicHU1MG10MmY5S1JX?=
 =?utf-8?B?Qy9XOVF2N0lNUWVIRk1Gd2xkb3EwbzY3ckY1RS9WMVNCSzIyRG1tdnBGdCtN?=
 =?utf-8?B?Rlo3WDlBSE5vS0FoWVE0aUpmeHU5bkwzK3RHbW96TWtFZzVYZHVnV2hPdU13?=
 =?utf-8?B?Wm5pMFljWVJlRFF6Ui9xdUZmc2VHVU5tSzZ1a3V0QWZ2L1ZicEc0ZUc4MmNL?=
 =?utf-8?B?L3RGU095RzE3N0dyY3ZsSWNFaTI3N0pLa0c0eXlNTDVDTFd4b3ZkTnFDL3JC?=
 =?utf-8?B?L2NCYm54VHpaUjYzZTBRbnVNdnBZckdGaXZmNE9mKy94eElONnZYNW5pT29i?=
 =?utf-8?B?TnhMcC9kYlU2MGM1TDJsTjVkTGN0Qnk0bFNGNWxrdDJjeDhoOFlJbjZxMDY4?=
 =?utf-8?B?aDNaY1MvYWNDaWp6V1NiMlA2YjBmdlltTjFLak9hTk9GQk1FZ21NZzNoWnZl?=
 =?utf-8?B?cENjWUFNTHBHM1FtU1VZQzNlb0RwYnBqL2JLWXU0cHVKNFBzY1FhSkg4citG?=
 =?utf-8?B?N1ZpTFFaaW5yYTdkWVdUd3A5NzFYaEhFUnpxaUcrbDhPbUhCazkySzB6Tmk4?=
 =?utf-8?B?MFlaRUV1Zm44eGFiWHNZa29nTGNZTEZKT1pHWmNoZFhPbHQzU2ZKOU5lekRG?=
 =?utf-8?Q?8Xf4duUMZA6v01udiUa9AhBu9?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cf09c95d-00ff-479d-e06c-08db0d71c16f
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2023 03:24:03.4305
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YWgbaaDlNNXFVcX/QrkgKfvZ0pwkTVEiBkkVX8uLagOPqZXzlnWTnHm/nvpBKAxVz8nELzkI05mKhj7bcNVUIA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4257
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2/13/2023 8:26 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
> 
>> On 2/8/2023 11:33 PM, Peter Zijlstra wrote:
>>> On Wed, Feb 08, 2023 at 01:05:28PM +0530, Bharata B Rao wrote:
>>>
>>>
>>>> - Hardware provided access information could be very useful for driving
>>>>   hot page promotion in tiered memory systems. Need to check if this
>>>>   requires different tuning/heuristics apart from what NUMA balancing
>>>>   already does.
>>>
>>> I think Huang Ying looked at that from the Intel POV and I think the
>>> conclusion was that it doesn't really work out. What you need is
>>> frequency information, but the PMU doesn't really give you that. You
>>> need to process a *ton* of PMU data in-kernel.
>>
>> What I am doing here is to feed the access data into NUMA balancing which
>> already has the logic to aggregate that at task and numa group level and
>> decide if that access is actionable in terms of migrating the page. In this
>> context, I am not sure about the frequency information that you and Dave
>> are mentioning. AFAIU, existing NUMA balancing takes care of taking
>> action, IBS becomes an alternative source of access information to NUMA
>> hint faults.
> 
> We do need frequency information to determine whether a page is hot
> enough to be migrated to the fast memory (promotion).  What PMU provided
> is just "recently" accessed pages, not "frequently" accessed pages.  For
> current NUMA balancing implementation, please check
> NUMA_BALANCING_MEMORY_TIERING in should_numa_migrate_memory().  In
> general, it estimates the page access frequency via measuring the
> latency between page table scanning and page fault, the shorter the
> latency, the higher the frequency.  This isn't perfect, but provides a
> starting point.  You need to consider how to get frequency information
> via PMU.  For example, you may count access number for each page, aging
> them periodically, and get hot threshold via some statistics.

For the tiered memory hot page promotion case of NUMA balancing, we will
have to maintain frequency information in software when such information
isn't available from the hardware.

Regards,
Bharata.