Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.161 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C
Message-ID: <49ed07b1-e167-7f94-9986-8e86fb60bb09@nvidia.com>
Date:   Wed, 9 Nov 2022 19:11:41 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: [PATCH v5 0/2] sched/numa: add per-process numa_balancing
Content-Language: en-US
To:     Gang Li <ligang.bdlg@bytedance.com>
CC:     <linux-api@vger.kernel.org>, <linux-kernel@vger.kernel.org>
References: <20221027025302.45766-1-ligang.bdlg@bytedance.com>
From:   John Hubbard <jhubbard@nvidia.com>
In-Reply-To: <20221027025302.45766-1-ligang.bdlg@bytedance.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Nov 2022 03:11:56.7561
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: d018ca13-2b54-4190-86c5-08dac2c95363
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT032.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5190
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/26/22 19:53, Gang Li wrote:
> # Introduce
> Add PR_NUMA_BALANCING in prctl.
> 
> A large number of page faults will cause performance loss when numa
> balancing is performing. Thus those processes which care about worst-case
> performance need numa balancing disabled. Others, on the contrary, allow a
> temporary performance loss in exchange for higher average performance, so
> enable numa balancing is better for them.
> 
> Numa balancing can only be controlled globally by
> /proc/sys/kernel/numa_balancing. Due to the above case, we want to
> disable/enable numa_balancing per-process instead.

Hi Gang Li,

Wow, it feels like I'm getting a Christmas present early, this is great
news!

This feature is something we've always wanted for GPUs and Compute
Accelerators, too. Because what happens there is: we might have GPUs in
the system that have mapped CPU memory into their page tables. When
autonuma unmaps the CPU ptes, this triggers (via mmu invalidate
callbacks) an unmapping on each GPU. But GPU mapping and unmapping is
far heavier weight than the corresponding CPU operations.

And so for things such as OpenCL apps that run on a GPU, the only viable
approach is to somehow disable autonuma balancing. And until your series
here, that was a system wide setting, which leads to not being able to
ever have things set up "right", without constantly intervening at the
sysadmin level.

So for the series, please feel free to add:

Acked-by: John Hubbard <jhubbard@nvidia.com>

thanks,
-- 
John Hubbard
NVIDIA

> 
> Set per-process numa balancing:
> 	prctl(PR_NUMA_BALANCING, PR_SET_NUMA_BALANCING_DISABLE); //disable
> 	prctl(PR_NUMA_BALANCING, PR_SET_NUMA_BALANCING_ENABLE);  //enable
> 	prctl(PR_NUMA_BALANCING, PR_SET_NUMA_BALANCING_DEFAULT); //follow global
> Get numa_balancing state:
> 	prctl(PR_NUMA_BALANCING, PR_GET_NUMA_BALANCING, &ret);
> 	cat /proc/<pid>/status | grep NumaB_mode
> 
> # Unixbench multithread result
> I ran benchmark 20 times, but still have measurement error. I will run
> benchmark more precisely on the next version of this patchset.
> +-------------------+----------+
> |       NAME        | OVERHEAD |
> +-------------------+----------+
> | Dhrystone2        | -0.27%   |
> | Whetstone         | -0.17%   |
> | Execl             | -0.92%   |
> | File_Copy_1024    | 0.31%    |
> | File_Copy_256     | -1.96%   |
> | File_Copy_4096    | 0.40%    |
> | Pipe_Throughput   | -3.08%   |
> | Context_Switching | -1.11%   |
> | Process_Creation  | 3.24%    |
> | Shell_Scripts_1   | 0.26%    |
> | Shell_Scripts_8   | 0.32%    |
> | System_Call       | 0.10%    |
> +-------------------+----------+
> | Total             | -0.21%   |
> +-------------------+----------+
> 
> # Changes
> Changes in v5:
> - replace numab_enabled with numa_balancing_mode (Peter Zijlstra)
> - make numa_balancing_enabled and numa_balancing_mode inline (Peter Zijlstra)
> - use static_branch_inc/dec instead of static_branch_enable/disable (Peter Zijlstra)
> - delete CONFIG_NUMA_BALANCING in task_tick_fair (Peter Zijlstra)
> - reword commit, use imperative mood (Bagas Sanjaya)
> - Unixbench overhead result
> 
> Changes in v4:
> - code clean: add wrapper function `numa_balancing_enabled`
> 
> Changes in v3:
> - Fix compile error.
> 
> Changes in v2:
> - Now PR_NUMA_BALANCING support three states: enabled, disabled, default.
>    enabled and disabled will ignore global setting, and default will follow
>    global setting.
> 
> Gang Li (2):
>    sched/numa: use static_branch_inc/dec for sched_numa_balancing
>    sched/numa: add per-process numa_balancing
> 
>   Documentation/filesystems/proc.rst   |  2 ++
>   fs/proc/task_mmu.c                   | 20 ++++++++++++
>   include/linux/mm_types.h             |  3 ++
>   include/linux/sched/numa_balancing.h | 45 ++++++++++++++++++++++++++
>   include/uapi/linux/prctl.h           |  7 +++++
>   kernel/fork.c                        |  4 +++
>   kernel/sched/core.c                  | 26 +++++++--------
>   kernel/sched/fair.c                  |  9 +++---
>   kernel/sys.c                         | 47 ++++++++++++++++++++++++++++
>   mm/mprotect.c                        |  6 ++--
>   10 files changed, 150 insertions(+), 19 deletions(-)
>