Message-ID: <a0127c3c-22c9-ac46-1e9f-200978fec392@amd.com>
Date:   Thu, 9 Feb 2023 11:27:40 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@suse.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        x86@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        tglx@linutronix.de, yue.li@memverge.com,
        Ravikumar.Bangoria@amd.com, ying.huang@intel.com
References: <20230208073533.715-1-bharata@amd.com>
 <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
Content-Language: en-US
From:   Bharata B Rao <bharata@amd.com>
In-Reply-To: <Y+Pj+9bbBbHpf6xM@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bllrdERNZDFxVTJtTXhCV2w0a2VUa1E5M1ExYnE1VTJRL3lxaFdiSzhWUy9T?=
 =?utf-8?B?SitkdVY5YWwxM0E3WEMray9wcHJtTlhDdGFSK3Nsa3BjcG5qRFhnUnZGVWZP?=
 =?utf-8?B?NlRRaG1NSXpMS2Z1SzBrMFd6cDh0TlFPYkN4U3gybjBaUzRza3liWlZ0MmNj?=
 =?utf-8?B?U0JFQzBidktYNXduWXhtVXl0L2hoZ0dRRHhZRG9qYS9YOXl1NWM0cDFHb2dD?=
 =?utf-8?B?Zjh4N21hT1QxYkxwVlNJcWd5a2p6ajNVM2txVU9CMzkwYkhBZWUzUWxIelhq?=
 =?utf-8?B?em5YdTNLTGFERHJQaklMeTVPSkd2Y2tUNGE0NHc0eDRiaHhnVVdXZTQyTHBO?=
 =?utf-8?B?SHcyMjYwc01EOEo4UzBnTUhuellFNVF3R3lKcVBFcHJoMHBzK1RBMGpJRlV4?=
 =?utf-8?B?ZEJobFFBdDVUY2o4dzBicFNnbkp0UUVEUVRHV3VLMHJIZEFlaWJzRTUrZ3RE?=
 =?utf-8?B?T0tUOE9KdzU3d2UzaTRmckJSbjloVzBZdlFhUnFXU09rRndKcVVSNFJVbGhI?=
 =?utf-8?B?MHZJNmx2eW4vcG5zWW5UellRa0J4ZXk2cllGMEJBS2FaM3dwSFB4ZlRWTGFW?=
 =?utf-8?B?cVdiR1BlZU1UOHJXU1hPam1oL2ZTQi8wN3RmODI3QnI3Mzl0N0djQUpiVWJl?=
 =?utf-8?B?U3o1UGd2MUJZbFhCbWkrNERldU5QWmtiS25XTTI2RnR2QjExK2NOTmNzY3Ry?=
 =?utf-8?B?QnBSQ0d5T2RCN1lRazA5K256L3JJNE9kMEJXYm5SQTFqelJQV0NNWXZRNnVn?=
 =?utf-8?B?WUE5VUdMWGF6eFFsQ3dhZ2YzcUsxdFdud2JnTVdIdlBoclFra0MxeWd2N0JH?=
 =?utf-8?B?ZHlzSzdEQTBucVFsRzZCQ1ROWDI5NUQ4ckU0RlBjZDl3bmt2cXN6OEthQnZG?=
 =?utf-8?B?VmpPd2JqcDZ6dXV1ZUFSTTh6TW9RSmpsekQxb0kwaWZPcFBPbmE5QmNUUnd3?=
 =?utf-8?B?djdCU1YxbWVJM0VVMjcxa0tHOUVhRW5kUzMwenV4clFrU0ZpRFJPQ0hJZVFx?=
 =?utf-8?B?L3dSeEc5LzdSTnZFYnRSTDJHKzlqcTNLZk5maDJmU1pOTDFqcm9ZNWs4eWhl?=
 =?utf-8?B?ZFhJUUVXaVFUV1h5dHo3eFU5aXMzMkZoaWM0QWE4MnVVSHYrdFJZaE9tQStI?=
 =?utf-8?B?Z3ZyREN0Rjg2ZityWHJpYnFqeDBCS1FoNDVPWW52MmNjRUxzVXEyYlJIcXla?=
 =?utf-8?B?YU1FV3U2aDQzRnlsSUVmSXBVVXZSdVdQZlVtemp0cGhSN3B3cWNwcnlhZ0NF?=
 =?utf-8?B?RnNkLzF4U0ZiSjJFeG95YVlrdzJKNzRMSkJaNkIrTFlCb1JwSWxqZlZycWFS?=
 =?utf-8?B?OGU2Y1JLMDFlUndTdnFXY1VWRjgyeUFCU1JnZ2R3Q0ViOFhvWkVWWms4bzBZ?=
 =?utf-8?B?OUNJM3Y4OTMrQmxDTThpM2RCaXhyUlloZjJoTGJJSEtKclhOL3NkQ0xENlJU?=
 =?utf-8?B?TUk4K3NuZ3N0UVZ6ZVdyL3lwSUc2Umo2dkhVMGROYTc3cTV6VzZ0STRWUUR0?=
 =?utf-8?B?OThmNjZQSVJYbnNvQVlLbjdVUVh0Zlg5WUNaRFVHa0dmR1FIYzRPckhza3F5?=
 =?utf-8?B?aHhqQjRqcHZ5NXl3SHdKaTVrbThFS0pwcEdXeEs2R3JGY2pOTzBZWUpoc1Bu?=
 =?utf-8?B?TzZlOFlYYWtZc3pqM21EY3l1ZWNTMXFvbUZXSVd3NHNRNDNlMGQxeE95NS9n?=
 =?utf-8?B?TTAzWDdGSG9UVUlTUW9KWmNNd09Dc1dkSVVGTlRyNFdVM1BYNVZJRjNaQTVF?=
 =?utf-8?B?bzlZMmNhcU9UN1E0WjZMeDBpS2IzdFE1bHFFdGZUNTg5bUlvWWRkbmFZcUV4?=
 =?utf-8?B?T1lMNFdkZ2Q1V0JxcndSSDhPSnFsSkhSQmZsVklhUnVIOGk5ems2cjZ1MUtI?=
 =?utf-8?B?bk9nbUs0TVNuY2xqSGVVaHloYk82QTZNTnRmOWR3cGM0QjRrbzEvTlpXZ1hQ?=
 =?utf-8?B?KzVpNUVJZWE4ZFJveXh0Q2IxV2FPeVllbG0rTWZRTXBFR1NCNEp0WSt0Q3lC?=
 =?utf-8?B?Tk1HTTZLeTJGZlhOYk1WYUR2STNWeDZkSGVlN01SL2RDb29hc3dDMStmVU5q?=
 =?utf-8?B?R1hpcjhNVHNkZWdyZ2sxT1dwYmNHOWxWVXAvRnAvYy91eUszK1BIZDJ2Ymg1?=
 =?utf-8?Q?8N1xUv6MiW3uVgO0XiKu5OZbL?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3265db30-a572-4a4a-a3fd-08db0a62943e
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2023 05:57:51.6543
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: B0P9E+2UM1clqEBb+9R20EXXF3+kmA5nYjVAB0fj8b/bOl67J/PxsFwZrh/BS9jfp8RqF6m/D2kQSB5ZoSfu3A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5368
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2/8/2023 11:33 PM, Peter Zijlstra wrote:
> On Wed, Feb 08, 2023 at 01:05:28PM +0530, Bharata B Rao wrote:
> 
>> - Perf uses IBS and we are using the same IBS for access profiling here.
>>   There needs to be a proper way to make the use mutually exclusive.
> 
> No, IFF this lives it needs to use in-kernel perf.

In fact I started out with in-kernel perf by using the
perf_event_create_kernel_counter() API. However there are issues
with using in-kernel perf:

- We want to reprogram the counter potentially during every
context switch. The IBS hardware sample counter needs to be
reprogrammed based on the incoming thread's view of sample period.
Additionally sampling needs to be disabled for kernel threads.
So I wanted to use perf_event_enable/disable() and
perf_event_period(). However they take mutexes and hence it is
not possible to use them from the sched switch atomic context.

- In-kernel perf gives a per-cpu counter, but we want it to count
based on the task that is currently running. I,e., the period should
be modified on per-task basis. I don't see how an in-kernel perf
event counter can be associated with per-task like this.
	
Hence I didn't see an easy option other than making the use
of IBS in perf and NUMA balancing mutually exclusive.

> 
>> - Is tying this up with NUMA balancing a reasonable approach or
>>   should we look at a completely new approach?
> 
> Is it giving sufficient win to be worth it, afaict it doesn't come even
> close to justifying it.
> 
>> - Hardware provided access information could be very useful for driving
>>   hot page promotion in tiered memory systems. Need to check if this
>>   requires different tuning/heuristics apart from what NUMA balancing
>>   already does.
> 
> I think Huang Ying looked at that from the Intel POV and I think the
> conclusion was that it doesn't really work out. What you need is
> frequency information, but the PMU doesn't really give you that. You
> need to process a *ton* of PMU data in-kernel.

What I am doing here is to feed the access data into NUMA balancing which
already has the logic to aggregate that at task and numa group level and
decide if that access is actionable in terms of migrating the page. In this
context, I am not sure about the frequency information that you and Dave
are mentioning. AFAIU, existing NUMA balancing takes care of taking
action, IBS becomes an alternative source of access information to NUMA
hint faults.

Thanks for your inputs.

Regards,
Bharata.