Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 23 Jun 2023 22:33:23 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Vincent Guittot <vincent.guittot@linaro.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        Abel Wu <wuyun.abel@bytedance.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        Len Brown <len.brown@intel.com>,
        Chen Yu <yu.chen.surf@gmail.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 3/4] sched/fair: Calculate the scan depth for idle
 balance based on system utilization
Message-ID: <ZJWtM1rgC52twqPS@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1686554037.git.yu.c.chen@intel.com>
 <bc96b53efb712337a645e2c2a340975545ed5a28.1686554037.git.yu.c.chen@intel.com>
 <20230621111721.GA2053369@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230621111721.GA2053369@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?T9Ot8hXCDLsEj3GDHhFSkrndhwIuWUqi41paeA1P3BgjqAQlLEZIO5g1chxr?=
 =?us-ascii?Q?65qW+Hmk7QG1h/3RAB1N6cBPU6yIf8LYlRxqwescYvzSjrv+9ArSEHGm7tTY?=
 =?us-ascii?Q?NO0knDCNUAyM//fn1u2MJC82HrQGsxvX1SkU9oA8xXvFcz4kQoj5CDBTSfXE?=
 =?us-ascii?Q?wa94Nw1D05jdEdk1DJAVc8wm5hhkhkIavMiev+J4BR1JT7ma/Auugjm7+OHa?=
 =?us-ascii?Q?2khvuoyU2CnpPrxBsnlGPQYcA6coqENc27m8X2V1r+zYYxgPCAaTeE5DOsPd?=
 =?us-ascii?Q?5b+KELAtR5IhPOXrgNxaU/NKKDjyHbGVmRemhEI7eCLkD4br+N9/lz7AvKIo?=
 =?us-ascii?Q?6xIfSOp1WkT12+AwyOg13kDytrK3kQlpQGDMNW4ayztwm4HkxCSvvIPSo7FI?=
 =?us-ascii?Q?pBRMdxIoiT2D1/xOJav3hMSe566ADTQWLPKUph6bGGAsWM4Idewj65HriSiB?=
 =?us-ascii?Q?qTERNloEBrJPFYvdftAq/Pe2QLANS0GN6BMaaFDJ/PS9s414X29HBMfPL9Bc?=
 =?us-ascii?Q?VYR5AjPfUMEXRUUMbE/U66hlJnZyt9EX2EffWoyPubYJ+uMnnDeuwQjUkaa8?=
 =?us-ascii?Q?15LNZXFAgUZmjxMD4/18RP39ADQkOnV3N4wVAVg5Zgj9xoI+V/4O6w/ioXEk?=
 =?us-ascii?Q?yK8WKtaWxIy25K8/synirbKJNvmgm8EBPMHmGS0oegUrz2lH1++z1LLgo3PD?=
 =?us-ascii?Q?n5bM0r/6AbbRxAmZ9N7Olny15HdBAZHvACUzR5x6vWhX1ifj3uMPJxHv3dc5?=
 =?us-ascii?Q?j4NwVL+MgeFQ8khTMufExDun0sLJBD9KS08R1+KgLx1DCWxMus02LUuunoOe?=
 =?us-ascii?Q?vuIm+Cpp31toIJbRiS3YxwddMi/mD7TEleaqoJhNpg4RfKrR0BMd/kzKfF36?=
 =?us-ascii?Q?PzEmUt90B/dzL2Om5h7J1WnK6Q2bWv7LzWGGJ4dZz3C1LnDd6VdQCTUvpY0t?=
 =?us-ascii?Q?sFSmSjKzrIPZPse5CYsUX0cP81nA+yAKcOH7q8k+jw2QUKekovZ9YV/KDWEw?=
 =?us-ascii?Q?5839O+C5RuzMKS8N4YhtRvMEMTZw67c7b3o2mTxHtbs4EtRZREKpmryQ0dsv?=
 =?us-ascii?Q?pgRpPrMUQcjY1P6kC1B8+I+RP+2L6Zwt3jNVoQZLpu+iYKmhrUF0DrSDwSSA?=
 =?us-ascii?Q?DAujuelqIFRgfWdSd5cZH6LlprIt2Oiv01XEy3DYQEpNHx4DKRW8wZYtOWbX?=
 =?us-ascii?Q?y4+K23mmdVQW3W3F9flCDbbyUGnUsbfARLf5GmHAvyJSdGs+l9deIzLG3C+h?=
 =?us-ascii?Q?zoNLw3h3wIOI7iOnq3bcA34ZrEyV7HNkR9JdPmGHVQf+camzGvq4f3qrNT3X?=
 =?us-ascii?Q?FZWoFwJX8eXm928UNBY0eFyiBxWdLfdybMjFvuR5RHEnw61OOMzlDyvOfrzu?=
 =?us-ascii?Q?o8Dx95YBUOWQXrnW0iG8p3iRnSVv+D+534tZvZDgvuWq5EKh2QpLk7hQ8D0I?=
 =?us-ascii?Q?8mJVrMqH4KPFEJgUw8AUAfO59rmm2w24M+BvRptnTTlz4H3L7qvjl8fY7zDD?=
 =?us-ascii?Q?GV8LlUl+z6btqsnLnWQ/CcutEUdEq0ZxOeaqa2F05KNiN8zyX2VylkMc5HnB?=
 =?us-ascii?Q?5cMoYyyylvlkefaYjbFrYv5NwfMukZHfNnqjfVj8?=
X-MS-Exchange-CrossTenant-Network-Message-Id: af35ee58-9b5d-4377-3e76-08db73f6d664
X-MS-Exchange-CrossTenant-AuthSource: CY5PR11MB6211.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2023 14:33:40.1107
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iu6V9EmMX0hQuWIVwETe/VUi86G3HKmW0Yqm5DzJwneqgqDHuUZoRZDZDND7cG+x0kUB4BSwy+IJP4hoUZ6bbQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR11MB6229
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,
On 2023-06-21 at 13:17:21 +0200, Peter Zijlstra wrote:
> On Tue, Jun 13, 2023 at 12:18:57AM +0800, Chen Yu wrote:
> > When CPU is about to enter idle, it invokes newidle_balance() to pull
> > some tasks from other runqueues. Although there is per domain
> > max_newidle_lb_cost to throttle the newidle_balance(), it would be
> > good to further limit the scan based on overall system utilization.
> > The reason is that there is no limitation for newidle_balance() to
> > launch this balance simultaneously on multiple CPUs. Since each
> > newidle_balance() has to traverse all the CPUs to calculate the
> > statistics one by one, this total time cost on newidle_balance()
> > could be O(n^2). This is not good for performance or power saving.
> 
> Another possible solution is to keep struct sg_lb_stats in
> sd->child->shared (below the NUMA domains) and put a lock around it.
> 
> Then have update_sd_lb_stats() do something like:
> 
> 	struct sg_lb_stats *sgs = &sds->sgs;
> 
> 	if (raw_spin_trylock(&sds->sg_lock)) {
> 		struct sg_lb_stats tmp;
> 
> 		... collect tmp
> 
> 		sds->sgs = tmp;
> 		raw_spin_unlock(&sds->sg_lock);
> 	}
> 
> 	... use sgs
> 
> Then you know you've always got a 'recent' copy but avoid the concurrent
> updates.
Thanks for taking a look and gave the suggestions! Yes, this is a good idea, by
doing this we can further limit the number of CPU to do update in parallel, and
allow the newidle CPU to reuse the data for idle load balance from others.
This lock only allow 1 CPU in that domain to iterate the whole group, and the
bottleneck might reply on how fast the CPU who grabs the lock can finish
collecting the tmp sgs data. For MC domain, it would not take too much time, and for
higher domains between MC and NUMA domain, it depends on how many CPUs there are in that
domain. I'll create one prototype based on your suggestion and measure the test data.

thanks,
Chenyu