Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 15 Jun 2023 09:52:46 +0530
From:   "Gautham R. Shenoy" <gautham.shenoy@amd.com>
To:     Chen Yu <yu.c.chen@intel.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        Abel Wu <wuyun.abel@bytedance.com>,
        Len Brown <len.brown@intel.com>,
        Chen Yu <yu.chen.surf@gmail.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        linux-kernel@vger.kernel.org
Subject: Re: [RFC PATCH 0/4] Limit the scan depth to find the busiest sched
 group during newidle balance
Message-ID: <ZIqSFjOcDvD5WZ2F@BLR-5CG11610CF.amd.com>
References: <cover.1686554037.git.yu.c.chen@intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <cover.1686554037.git.yu.c.chen@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?HcG4AkopdiwdITJa573QHXITytNmr8xUNvyKEVbbWqZOR3oxEgss+KNMfarZ?=
 =?us-ascii?Q?cVHJTY4rJtI4hUPqZng1G/QVoYWGNGBISMlu6icI8fhQITOX83OZhC2J+Iu1?=
 =?us-ascii?Q?Fq1oKdaCR9bPqEV+lCMRk7mhGlfQMr7VMQtNGXIO6SLhBGaf0ch9V9HMaXPr?=
 =?us-ascii?Q?jRhJnZTynod8xpbD50lIsSHYr/D5bKhEaPGDarrFmKfx+DLQ8QnJ1WYN2JAv?=
 =?us-ascii?Q?ZSmn4JfVHL4DAHy4JoAL/5btN7PQcQH7jay8Hrggz87oR8KV/uav/uOw8tLs?=
 =?us-ascii?Q?Ie4E65+YOVuswj3GusIvBtIzauGRLx4uOKreGd5I8uQ0q/lfYsQYcrgwU8OZ?=
 =?us-ascii?Q?OI/bbo96GgGDgR4GbsXXfsEAsqWd8U4M6oruZkek5iW/ipWUyN5GkG0D4FJ7?=
 =?us-ascii?Q?PiU019nKdVx4V7/2iLqcSzIOLhFaQauqkKbpoGIWYbmRWg3vYbpsVYX2YlZ3?=
 =?us-ascii?Q?ZHfb+JzNBfgvrWo/C6IzJr77INqVAdYbR764c6Inwk2PjTeI5zQqI35yXSfa?=
 =?us-ascii?Q?lpGVizpukoEqf8HV75iqQIA+ZaVFjwU0tZ55hb5qZY+SDsplcHlL2MJ8rAHp?=
 =?us-ascii?Q?qL7MjSd0DvNe1SfC+W151MNcUTrk5/4GbK6jWO+o3Dl7pJfwc23qVXfLU74p?=
 =?us-ascii?Q?Fh07Dzf7TDjQ0/7kNbkrke4//VPYfQ+8QBPQMWVx31Q9uT7z17cM2Ssjnncr?=
 =?us-ascii?Q?/TJoGjzWNtz6P9A+PR6KSnz5p/UvEJ3XKwq5VrpCRPvTp9jkvUIulVPIS6WH?=
 =?us-ascii?Q?O7U6Qxu011d+HKFRFwZsQzYeVjzxDh9XCQ7kdAmPGTUY2IC3HEavHGVJJw+B?=
 =?us-ascii?Q?34WLk74uKFwUfDgBn07Caswiq9YRmUCUBzgTa2NExojTO4JqP4J1TLEUr0ar?=
 =?us-ascii?Q?PNa99pVl/XWuuNnbE+NVOi47fSq4IUaQHqYA2ughY6s3+K9t/ocZ9Q/x8c/+?=
 =?us-ascii?Q?uPqxlByeOYxg+PwLZkDevd5PIqrlueeaoRah6y21VIx854O6CN4EnuymYPT+?=
 =?us-ascii?Q?hQf3qu5BwNQjg9Da3toReGlSpvqQHkKX50ndsL7uGZYVw4rOCpp24WWPBiQT?=
 =?us-ascii?Q?+hnmjLyFx4dGIWM8shqt64Rbd7/F8U/Kr8VJ+zBb38wHPZoBJey6XMroBHHA?=
 =?us-ascii?Q?EEscOeaeczW3haFKQSYNobmmtGDkII5852xLWrdMweGDN5nj2ryXtB1Emt7K?=
 =?us-ascii?Q?URHJQH2IMzrOLophbAl5gIxAIpQ/MweD3ZItBzeP5TF7V+ewSZ8QbNyHIqB1?=
 =?us-ascii?Q?VqGjMRUYsPUxUueOV2BWizDGapWclRFG2nKT8lQxODm9JoIdRXjUGV9RQZGY?=
 =?us-ascii?Q?5uNdXHyQNGM06bbpr0/zh22kKB99qQoj26M6qRnUYOo11qYLHo2YeNW9C9Do?=
 =?us-ascii?Q?+KJbzGo9v08JOkr5HNPBkangGUNNOUhxQtVB5VsH2NgRec2OKt3M8+nkTv1n?=
 =?us-ascii?Q?qhhSoKCV8otfbNyIqboMnMttMPvwLNtEPeO0Piq09167W0h0FORx9Lqxi/QN?=
 =?us-ascii?Q?QDT0RT2dK6Mr2wEWI+cEzMBoD5I65AN6vaFgCo8/A0Ye/4rANnGsqPFsJgX2?=
 =?us-ascii?Q?pwzw70P9kclHIvM1mGDlkXbPxlOOXSgLizK0SaJk?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 18fd71c2-c164-4368-9915-08db6d583387
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3286.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jun 2023 04:22:59.4317
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RWA/an7WY1XVVa6bpLJjX56r00tPSNv+JG6jogQVgkNIEutZX0ycCn7/yxw1DNvkE39yWynS2A3L5ClckRmH6A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7342
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello Chen Yu,


On Tue, Jun 13, 2023 at 12:17:53AM +0800, Chen Yu wrote:
> Hi,
> 
> This is an attempt to reduce the cost of newidle balance which is
> found to occupy noticeable CPU cycles on some high-core count systems.
> 
> For example, by running sqlite on Intel Sapphire Rapids, which has
> 2 x 56C/112T = 224 CPUs:
> 
> 6.69%    0.09%  sqlite3     [kernel.kallsyms]   [k] newidle_balance
> 5.39%    4.71%  sqlite3     [kernel.kallsyms]   [k] update_sd_lb_stats
> 
> The main idea comes from the following question raised by Tim:
> Do we always have to find the busiest group and pull from it? Would
> a relatively busy group be enough?
> 
> The proposal ILB_UTIL mainly adjusts the newidle balance scan depth
> within the current sched domain, based on the system utilization in
> this domain. The more spare time there is in the domain, the more time
> each newidle balance can spend on scanning for a busy group. Although
> the newidle balance has per domain max_newidle_lb_cost to decide
> whether to launch the balance or not, the ILB_UTIL provides a smaller
> granularity to decide how many groups each newidle balance can scan.

Thanks for the patchset. This is an interesting approach to minimise
the time spent in newidle balance when the system is relatively
busy. In the past we have seen that newly idle CPUs whose avg idle
duration is slightly higher than the max_newidle_balance_cost spend
quite a bit of time trying to find a busiest group/cpu only to not
find any task to pull. But that's the opportunity lost to go idle. I
suppose this patch is targetted towards a SMT --> MC(DIE) kind of
topologies where we have a lot of groups in the DIE domain, but it
would be interesting to see if this can help when there are fewer
groups in each sched-domain.

Will try this on our setup.

> 
> patch 1/4 is code cleanup.
> 
> patch 2/4 is to introduce a new variable in sched domain to indicate the
>           number of groups, and will be used by patch 3 and patch 4.
> 
> patch 3/4 is to calculate the scan depth in each periodic load balance.
> 
> patch 4/4 is to limit the scan depth based on the result of patch 3,
>           and the depth will be used by newidle_balance()->
>           find_busiest_group() -> update_sd_lb_stats()
> 
> 
> According to the test result, netperf/tbench shows some improvements
> when the system is underloaded, while no noticeable difference from
> hackbench/schbench. While I'm trying to run more benchmarks including
> some macro-benchmarks, I send this draft patch out and seek for suggestion
> from the community if this is the right thing to do and if we are in the
> right direction.
> 
> [We also have other wild ideas like sorting the groups by their load
> in the periodic load balance, later newidle_balance() can fetch the
> corresponding group in O(1). And this change seems to get improvement
> too according to the test result].
> 
> Any comments would be appreciated.
> 
> Chen Yu (4):
>   sched/fair: Extract the function to get the sd_llc_shared
>   sched/topology: Introduce nr_groups in sched_domain to indicate the
>     number of groups
>   sched/fair: Calculate the scan depth for idle balance based on system
>     utilization
>   sched/fair: Throttle the busiest group scanning in idle load balance
> 
>  include/linux/sched/topology.h |  5 +++
>  kernel/sched/fair.c            | 74 +++++++++++++++++++++++++++++-----
>  kernel/sched/features.h        |  1 +
>  kernel/sched/topology.c        | 10 ++++-
>  4 files changed, 79 insertions(+), 11 deletions(-)
> 

--
Thanks and Regards
gautham.