Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <6ef38c6e-e47d-66c2-216a-76ab4a59feb1@amd.com>
Date:   Mon, 23 Oct 2023 10:55:55 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [PATCH V1 0/1] sched/numa: Fix mm numa_scan_seq based
 unconditional scan
Content-Language: en-US
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org
Cc:     Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Mel Gorman <mgorman@suse.de>,
        Andrew Morton <akpm@linux-foundation.org>,
        David Hildenbrand <david@redhat.com>, rppt@kernel.org,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Bharata B Rao <bharata@amd.com>,
        Aithal Srikanth <sraithal@amd.com>,
        kernel test robot <oliver.sang@intel.com>,
        Sapkal Swapnil <Swapnil.Sapkal@amd.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>
References: <cover.1697816692.git.raghavendra.kt@amd.com>
From:   Raghavendra K T <raghavendra.kt@amd.com>
In-Reply-To: <cover.1697816692.git.raghavendra.kt@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OUl5cUgvcVNMVWh4b3dNSGNvb3VhQkZ2QlRrRG9IOWpCbmJwZ1laSlR0VzlO?=
 =?utf-8?B?M0tIZVZqdEJjME9vUEQ0dXVzSSswSjU1NnF6TnR3RGxpb1hZbjNTamtSMVFN?=
 =?utf-8?B?QkhBNVFWanhPZUNIaG1saXNaU3pxUk15Mk5ybFNGQ2hwOHBMRC9tTFB0UnJ5?=
 =?utf-8?B?NUg2STMvd3RRaDZ4RWx2L205ZGdsa1EyamQ1blpOUWNrUTd5akZSaVRkSlpp?=
 =?utf-8?B?MDRJVFBTMjNJYnozRDZEUWs1OCs0K3IxVDhxT1lmVkZSNGlZNU1LbG9RVm41?=
 =?utf-8?B?ci9ZNDZMU3hVeGRNaEloQmZQdFBrL3BRUUluZmNFdCtzdk80V2ZXN0V2RnN2?=
 =?utf-8?B?ZUh0ZXNRUDNQZ1ZZaHEwWDF6b0xENEpjUzBvUU0wR0g4TXI2QkdPL1R0QVR0?=
 =?utf-8?B?NWhrNWRLWk1kekx4akVsM1k1bWtWSk4rNERlZm5iOS8zVEJUZ3RBQWFnTDlu?=
 =?utf-8?B?eEJHMG5xSjl2ZnlUcGxPam0yY2ZqQWtETEVjYWEwYlI0WTVXZHp1WnpXWHdU?=
 =?utf-8?B?eW9WRVNTbWFUSktNTlI1TENqK0U5WlRYUGRpV0J2NlhBbXl4REZudTc3WTB2?=
 =?utf-8?B?SFdjdW9VNTlNdmdHNFFxT3lGbk92amVYZE1FcTlWOGJUN244bFgvcFVMN0ZQ?=
 =?utf-8?B?L1h3ZERxWit6UVdkcStNd2JCcnRmMEwxaUxhMEVKb3J6cWYrSVhoa0N2TXI4?=
 =?utf-8?B?SWYyWnZjMkdYVzlxK3FyYjVWaFBkQy9tNWN4dzBKKzl1N3FqcmZJVThTMTdX?=
 =?utf-8?B?aGNCYk8zVnJEL2xJVFUwUDcvVXRUYi9wOEZPdDcyMVFYUTI2dFlrR2Q3bm5I?=
 =?utf-8?B?SXdIUk55VmZjRHRaQk5FRiswZU1LcjhGVFgwSDloUkFxRjlUL2RVLzhjWGNz?=
 =?utf-8?B?S0hNMExPcGJsbnM1aGl5NVoxbUFodThtQnZ3ZStQZDErOXM0amlmZ0hVSFN0?=
 =?utf-8?B?dGdsaG8wSklRVVJNUTFwRE9nalNSdExoQ0xZNkpwRElTL0hYMU9EbDM1dXUz?=
 =?utf-8?B?bFIxTm1KTUVaYUhNSGZaWmlEZno2eStncE5PWjZDUmdZK21oT3JDTDc0ZCts?=
 =?utf-8?B?S2k2TzczZ0grek9GZ1dIRXJsdldpMjdFMUQ2Q1hyWGF0VklnZWZFZGtOMmdu?=
 =?utf-8?B?MzB6Ym5weitVQncwNjduaGFFWWwydUZlcld6d01yajRIVU5wNmtoZStjY1Zi?=
 =?utf-8?B?d0JiQWtweUg0VTVFWXNMN1l2V2RaVURUemtGQ1lod2I1ZS9EYmpJQndubjJQ?=
 =?utf-8?B?T05XbWVwZ2t2SXF3di9jZkFyRlhmUU9tVmZCeTVQVFEyMzNJQzBETXh1U0Jv?=
 =?utf-8?B?MUVDeHNqYWptdytoYlRmVVBTWWduMGhOc1h4eEMxWFgyUTU4OFNHb1lVa01G?=
 =?utf-8?B?Y2VsS1g2Z0RwN2RYdnhHQTNqRjVyY3ZRK1IzWDhkK0lKRVJXQU1RWVV3bHBX?=
 =?utf-8?B?S0NuSEs0L2h1WXM5dEpFRjRLNThXOEpheEtlWHN2MnNLVmNrYUhSVU4ybHRl?=
 =?utf-8?B?Zkg0N0d6RENsWWVCclVyeXNRMmFvQlRyeDlERFhrdFdPY0JyZSttb1lnK0RW?=
 =?utf-8?B?Z0VxVXV3TjZiUmpVYVkvRjN1RVpkUzZoUlUwSTF3S3ltSUZKK3NsS1gvdE1X?=
 =?utf-8?B?dmpOeHdKMUJvaHJacS9UMnFJSHdJUytpSHp3Y1YzSjlEZmtHVUFucys3UGRB?=
 =?utf-8?B?MTlSZ2pTTk56Mnk5bkhxRzhxdmtQams2dzRPdE9USXdYVk1RYVVUL1BQREtv?=
 =?utf-8?B?U0RSMWZlQ1pwYXIrd2JNWHVDMzNTQ0tuQk9IUW9TUzU3M0hIb1NUZHYxa0xy?=
 =?utf-8?B?OUV0Um8yazdwQ2FKSDVSWHRlWXBtN214NWQvaGpPS1BmdUlSbWFLb3BBU1ph?=
 =?utf-8?B?VWZLQm9pa0VzTnIwVkI3Y2lISm1CNkRUWFNlNzlvdWZwN1NDaUJ1aXZkNDlZ?=
 =?utf-8?B?WEZwL2E5dDhiUUdUYjVueVhxbjZKUU9RY0RpSzlEbkFCdkpMNnUwODlmR3ND?=
 =?utf-8?B?dXM2OHphN0dNTHJhZmdVUmo3ZnJjajV2cGVqQTZTT3dFZnpJOVVlazZlR3FL?=
 =?utf-8?B?RDdSUEMydzlBdjVPcmwrMGl5b0haMWZRTGJjRS9jMWdEbHJubVB0UFlxRXor?=
 =?utf-8?Q?80CNUJWMvBHJ/HnlJr/CJ27yz?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c58578b1-8b8b-4ac1-6b24-08dbd3888dcd
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3008.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2023 05:26:05.7836
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mD2ePBIlP0FcYuNH7q8qPwlhykxCqJWatVmzTAIuxgrfmSR6+5f6qnUZGA1e7UZpTRtKT7ECingF32+pLlL6Ww==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB8613
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 22 Oct 2023 22:26:16 -0700 (PDT)

On 10/20/2023 9:27 PM, Raghavendra K T wrote:
> NUMA balancing code that updates PTEs by allowing unconditional scan
> based on the value of processes' mm numa_scan_seq is not perfect.
> 
> More description is in patch1.
> 
> Have used the below patch to identify the corner case.
> 
> Detailed Result: (Only part of the result is updated
> in patch1 to save space in commit log)
> 
> Detailed Result:
> 
> SUT: AMD EPYC Milan with 2 NUMA nodes 256 cpus.
> 
> Base kernel: upstream 6.6-rc6 (dd72f9c7e512) with Mels patch-series
> from tip/sched/core [1] applied.
> 
> Summary: Some benchmarks imrove. There is increase in system
> time due to additional scanning. But elapsed time shows gain.
> 
> However there is also some overhead seen for benchmarks like NUMA01.
> 
> kernbench
> ==========		base                  patched
> Amean     user-128    13799.58 (   0.00%)    13789.86 *   0.07%*
> Amean     syst-128     3280.80 (   0.00%)     3249.67 *   0.95%*
> Amean     elsp-128      165.09 (   0.00%)      164.78 *   0.19%*
> 
> Duration User       41404.28    41375.08
> Duration System      9862.22     9768.48
> Duration Elapsed      519.87      518.72
> 
> Ops NUMA PTE updates                 1041416.00      831536.00
> Ops NUMA hint faults                  263296.00      220966.00
> Ops NUMA pages migrated               258021.00      212769.00
> Ops AutoNUMA cost                       1328.67        1114.69
> 
> autonumabench
> 
> NUMA01_THREADLOCAL
> ==================
> Amean     syst-NUMA01_THREADLOCAL       10.65 (   0.00%)       26.47 *-148.59%*
> Amean     elsp-NUMA01_THREADLOCAL       81.79 (   0.00%)       67.74 *  17.18%*
> 
> Duration User       54832.73    47379.67
> Duration System        75.00      185.75
> Duration Elapsed      576.72      476.09
> 
> Ops NUMA PTE updates                  394429.00    11121044.00
> Ops NUMA hint faults                    1001.00     8906404.00
> Ops NUMA pages migrated                  288.00     2998694.00
> Ops AutoNUMA cost                          7.77       44666.84
> 
> NUMA01
> =====
> Amean     syst-NUMA01       31.97 (   0.00%)       52.95 * -65.62%*
> Amean     elsp-NUMA01      143.16 (   0.00%)      150.81 *  -5.34%*
> 
> Duration User       84839.49    91342.19
> Duration System       224.26      371.12
> Duration Elapsed     1005.64     1059.01
> 
> Ops NUMA PTE updates                33929508.00    50116313.00
> Ops NUMA hint faults                34993820.00    52895783.00
> Ops NUMA pages migrated              5456115.00     7441228.00
> Ops AutoNUMA cost                     175310.27      264971.11
> 
> NUMA02
> =========
> Amean     syst-NUMA02        0.86 (   0.00%)        0.86 *  -0.50%*
> Amean     elsp-NUMA02        3.99 (   0.00%)        3.82 *   4.40%*
> 
> Duration User        1186.06     1092.07
> Duration System         6.44        6.47
> Duration Elapsed       31.28       30.30
> 
> Ops NUMA PTE updates                     776.00         731.00
> Ops NUMA hint faults                     527.00         490.00
> Ops NUMA pages migrated                  183.00         153.00
> Ops AutoNUMA cost                          2.64           2.46
> 
> Link: https://lore.kernel.org/linux-mm/ZSXF3AFZgIld1meX@gmail.com/T/
> 

Forgot to add skip_vma_count trace results:

autonumabench: numa01_THREAD_LOCAL 3 iterations

base:
inaccessible:13133
pid_inactive:15807
scan_delay:471
seq_completed:50
shared_ro:6983
unsuitable:3917

patched:
inaccessible:4727
pid_inactive:5119
scan_delay:455
seq_completed:7
shared_ro:2551
unsuitable:5402


> Raghavendra K T (1):
>    sched/numa: Fix mm numa_scan_seq based unconditional scan
> 
>   include/linux/mm_types.h | 3 +++
>   kernel/sched/fair.c      | 4 +++-
>   2 files changed, 6 insertions(+), 1 deletion(-)
> 
> ---8<---
> diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
> index 010ba1b7cb0e..a4870b01c8a1 100644
> --- a/include/trace/events/sched.h
> +++ b/include/trace/events/sched.h
> @@ -10,6 +10,30 @@
>   #include <linux/tracepoint.h>
>   #include <linux/binfmts.h>
>   
> +TRACE_EVENT(sched_vma_start_seq,
> +
> +	TP_PROTO(struct task_struct *t, struct vm_area_struct *vma, int start_seq),
> +
> +	TP_ARGS(t, vma, start_seq),
> +
> +	TP_STRUCT__entry(
> +		__array(	char,	comm,	TASK_COMM_LEN	)
> +		__field(	pid_t,	pid			)
> +		__field(	void *,	vma			)
> +		__field(	int, start_seq		)
> +	),
> +
> +	TP_fast_assign(
> +		memcpy(__entry->comm, t->comm, TASK_COMM_LEN);
> +		__entry->pid	= t->pid;
> +		__entry->vma	= vma;
> +		__entry->start_seq	= start_seq;
> +	),
> +
> +	TP_printk("comm=%s pid=%d vma = %px start_seq=%d", __entry->comm, __entry->pid, __entry->vma,
> +			 __entry->start_seq)
> +);
> +
>   /*
>    * Tracepoint for calling kthread_stop, performed to end a kthread:
>    */
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index c8af3a7ccba7..e0c16ea8470b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3335,6 +3335,7 @@ static void task_numa_work(struct callback_head *work)
>   				continue;
>   
>   			vma->numab_state->start_scan_seq = mm->numa_scan_seq;
> +			trace_sched_vma_start_seq(p, vma, mm->numa_scan_seq);
>   
>   			vma->numab_state->next_scan = now +
>   				msecs_to_jiffies(sysctl_numa_balancing_scan_delay);
> 
>