Message-ID: <8fea74ec-8feb-1709-14f2-cecb63fdc9ed@amd.com>
Date:   Fri, 24 Feb 2023 09:06:49 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
Content-Language: en-US
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@suse.de,
        peterz@infradead.org, mingo@redhat.com, bp@alien8.de,
        dave.hansen@linux.intel.com, x86@kernel.org,
        akpm@linux-foundation.org, luto@kernel.org, tglx@linutronix.de,
        yue.li@memverge.com, Ravikumar.Bangoria@amd.com
References: <20230208073533.715-1-bharata@amd.com>
 <878rh2b5zt.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <72b6ec8b-f141-3807-d7f2-f853b0f0b76c@amd.com>
 <87zg9i9iw2.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <1547d291-1512-faae-aba5-0f84c3502be4@amd.com>
 <c3e79d2e-97da-726e-bcaa-0258e3ddfafe@amd.com>
 <87zg9c7rrf.fsf@yhuang6-desk2.ccr.corp.intel.com>
From:   Bharata B Rao <bharata@amd.com>
In-Reply-To: <87zg9c7rrf.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RVNjNkFucGpaT1BuL2Y5QUt5dmtGRUhlS0Z6amJodm1oZSt6U1J2Q2lQcXAv?=
 =?utf-8?B?QlhKaktDcldRcVJhbitJVmlvNUs0UDh0RXBPRzRST0FRYkdwMlFneXpPQmYw?=
 =?utf-8?B?M3BkUWo3NStNeHBXcVVORUFpMFpQTWwveHNGZlY2Zm44RWlzUHRrak91VEdy?=
 =?utf-8?B?NXJTODQzWkZjN2xqTDYzRGkweTkvVTFJcFloZEthSzdab2s2KzJJaDhNVEMx?=
 =?utf-8?B?cDZmSGc0ZzFFT3JZeHVDdEY1WXpYWk9mczdpamx5dnNzL0lHZDRpV1huZnNR?=
 =?utf-8?B?dUdoazJBM1VsdUIxc2h2aEhreS9DS2ZobXdpSmlSOTR5cWhqMDM3NVNLUVJP?=
 =?utf-8?B?dlJuNWVBUlZNWCtGcHBMV3lWOUxSa2dpbElLSWFPQVNjbnB5MkdPVFRyYkRM?=
 =?utf-8?B?QnJLSXZncVhKYlJZWEc0b1labFF1R1E3a2NZTFN4dk5pTUtIM1VRT3pXQWdN?=
 =?utf-8?B?NGxleHZLb29wUlBZRVBjaUtHVjgyQjhjZ2JDc1FJM3RHMzFxdnY1OCtUQ080?=
 =?utf-8?B?eGJxNEFCaHNFTzduQklsVlN4cXF4QnQ5YURNQ2RIWDE0ZUhwanFhRTJLUDBy?=
 =?utf-8?B?VzFmR2l5aDErV0VzV3g3VEYweHFmNGMvQTVMMGsydjJ6MzYvRUl1MzB6RFNN?=
 =?utf-8?B?MU5xNld2bHRscmtKWTA1Y0JuWGZCWE9jL1doMTFtZ0dYOEVaYkNZT3Y4bGJ4?=
 =?utf-8?B?Y2lnWFR2dlNDUGdFVk1sNnFVSVBuRGlicElhVldSK3pqY202bnFnRSt4TmVH?=
 =?utf-8?B?eDloTU44YUFnQUY1R3FySjFJV1ZyalNBM0trbWdMTFcxbHNBa1loMEt0Z2pm?=
 =?utf-8?B?b3Rrd2dNZGljakF6bzF2Y1NnUnYxVmdWcmtHTnR6Nnl4L1Vzd3BNWkZ0SHRP?=
 =?utf-8?B?alRJWFVOd2JPTWVrTDk5SlV3NUlWNlVDbVdvN0MrdHZkL1BLLzB0RTdXeGcw?=
 =?utf-8?B?V2dNUW1yV1hlYWZtdjQwYzRqcVRidmtZTHl5NDJSMWJNV3N6U3B4dWhURWZD?=
 =?utf-8?B?OTlMQm1sdTBKQ2lvZ1hIY0xuU2VKYzk3ZkIzZWRqbTc1L2RWWTJrSHRld3Ji?=
 =?utf-8?B?L1RMR3RDc0FRYTA1aWQzanFXNWJRS0tmNUthVml6U2diU2RTaVNTL3Y1a2hI?=
 =?utf-8?B?azY2U2d2VUpDUlJVZ3UzWFR6NEkrVkFvV1VJVDQrQVRYRHNzSmd0V2E1TnVm?=
 =?utf-8?B?TlRJNTdtaU9OUjZTVTk5NVQrVnNTQVZwd0lHcVlwUjN4UGxkdmx4d3BraFo5?=
 =?utf-8?B?SXBOQlZDMksxR2lyczZiSXROYnp1ZlRQUWEyeHh1eU1YZ20vUlFMay9yV0ZI?=
 =?utf-8?B?ekp0RXAzNDZLdWxGQ2VwbXRZR09mc1ZpVStlem1FZFNJNUJpd1BDRHpwdUZT?=
 =?utf-8?B?eldEQUY3MmZnVlo2OVhobVF1eFo3bnE5QkJzeURTeUhPMVc5cE9IdDU1RlB6?=
 =?utf-8?B?a3BMQkpNKzF1YTVDQkV1MHZjdFhLVm9pb3FybjdGU1MxdkxnbDFaNUsvV2NH?=
 =?utf-8?B?Y0oxUnpOc2l3KzFsNERkWHdTbHg0MGtYMjl3NWpCeUJsY3FiN04vaTVuZXp5?=
 =?utf-8?B?UElZMXpRRkFpUE41Uk9EVDNBaStadytrNDROQU9NNExHRTJhUjBJb2ZUdW1p?=
 =?utf-8?B?aE9KZEZsN3lkc0tzalNYQ1QvNWJ5UEZJT2dNL3EzZlMwdTFTUWFMcGdzaExo?=
 =?utf-8?B?WGhlblJMMjlCeTZBenQ2YjNNZ3J4V0ZNRFA0TWxFeUdNdjFtNnY5NUgvclFK?=
 =?utf-8?B?VlhLSGpISTlqbm1rbm83UFQyQ3dXTGhPMHN1bGJ4aVdLMzhYZ1hpQlJWRnkz?=
 =?utf-8?B?cER6TE84ZWQ0TUs5RVhiTXRqZjVDVlhFR0lhbDZRNVdLNk55R25KeG5IUWVN?=
 =?utf-8?B?VG5WTFVEejBJT2drSHRhNnIzS2hTYmJsSVV4alRHSS9uR1REZUc2MnpDRmFX?=
 =?utf-8?B?RFFXRTRPem5LclVtbStKTzRlZjBnYURIbWhSV2FpT3c1ZG01RWNqc2VsazNr?=
 =?utf-8?B?Y1M0NUpLUjZDMHhNSWNQRk9EZXQ1T2dRbkVRWVQ2VHB0ZGhHYzBOQW13NlpW?=
 =?utf-8?B?Z3Q2SkRhSk45VWNNTnM0TXhHdXAyWmRxRjVuUDIvdlRkUm1icUo3RXdlM05U?=
 =?utf-8?Q?frrWGN2XffcvdAWHjF61Ny8At?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3aaab5fe-cc23-4559-6287-08db161863b6
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2023 03:37:01.4573
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RUq3gH9swJOwbzUcGKZlkQtSBPUHUYbXA7plRPKzScb47HJaNe4tmsIg7izKgA5gu79iRuUQIPk7i0AQezraGg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7639
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 17-Feb-23 11:33 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
> 
>> On 14-Feb-23 10:25 AM, Bharata B Rao wrote:
>>> On 13-Feb-23 12:00 PM, Huang, Ying wrote:
>>>>> I have a microbenchmark where two sets of threads bound to two 
>>>>> NUMA nodes access the two different halves of memory which is
>>>>> initially allocated on the 1st node.
>>>>>
>>>>> On a two node Zen4 system, with 64 threads in each set accessing
>>>>> 8G of memory each from the initial allocation of 16G, I see that
>>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time
>>>>> to complete a fixed number of memory accesses. This could well
>>>>> be the best case and real workloads/benchmarks may not get this much
>>>>> uplift, but it does show the potential gain to be had.
>>>>
>>>> Can you find a way to show the overhead of the original implementation
>>>> and your method?  Then we can compare between them?  Because you think
>>>> the improvement comes from the reduced overhead.
>>>
>>> Sure, will measure the overhead.
>>
>> I used ftrace function_graph tracer to measure the amount of time (in us)
>> spent in fault handling and task_work handling in both the methods when
>> the above mentioned benchmark was running.
>>
>> 			Default		IBS
>> Fault handling		29879668.71	1226770.84
>> Task work handling	24878.894	10635593.82
>> Sched switch handling			78159.846
>>
>> Total			29904547.6	11940524.51
> 
> Thanks!  You have shown the large overhead difference between the
> original method and your method.  Can you show the number of the pages
> migrated too?  I think the overhead / page can be a good overhead
> indicator too.
> 
> Can it be translated to the performance improvement?  Per my
> understanding, the total overhead is small compared with total run time.

I captured some of the numbers that you wanted for two different runs.
The first case shows the data for a short run (less number of memory access
iterations) and the second one is for a long run (more number of iterations)

Short-run
=========
Time taken or overhead (us) for fault, task_work and sched_switch
handling

			Default		IBS
Fault handling		29017953.99	1196828.67
Task work handling	10354.40	10356778.53
Sched switch handling			56572.21
Total overhead		29028308.39	11610179.41

Benchmark score(us)	194050290	53963650
numa_pages_migrated	2097256		662755
Overhead / page		13.84		17.51
Pages migrated per sec	72248.64	57083.95

Default
-------
			Total		Min	Max		Avg
do_numa_page		29017953.99	0.1	307.63		15.97
task_numa_work		10354.40	2.86	4573.60		175.50
Total			29028308.39

IBS
---
			Total		Min	Max		Avg
ibs_overflow_handler	1196828.67	0.15	100.28		1.26
task_ibs_access_work	10356778.53	0.21	10504.14	28.42
hw_access_sched_in	56572.21	0.15	16.94		1.45
Total			11610179.41


Long-run
========
Time taken or overhead (us) for fault, task_work and sched_switch
handling
			Default		IBS
Fault handling		27437756.73	901406.37
Task work handling	1741.66		4902935.32
Sched switch handling			100590.33
Total overhead		27439498.38	5904932.02

Benchmark score(us)	306786210.0	153422489.0
numa_pages_migrated	2097218		1746099
Overhead / page		13.08		3.38
Pages migrated per sec	6836.08		11380.98

Default
-------
			Total		Min	Max		Avg
do_numa_page		27437756.73	0.08	363.475		15.03
task_numa_work		1741.66		3.294	1200.71		42.48
Total			27439498.38

IBS
---
			Total		Min	Max		Avg
ibs_overflow_handler	901406.37	0.15	95.51		1.06
task_ibs_access_work	4902935.32	0.22	11013.68	9.64
hw_access_sched_in	100590.33	0.14	91.97		1.52
Total			5904932.02

Regards,
Bharata.