Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 10 Jul 2023 23:37:06 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     K Prateek Nayak <kprateek.nayak@amd.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Abel Wu <wuyun.abel@bytedance.com>,
        "Gautham R. Shenoy" <gautham.shenoy@amd.com>,
        Len Brown <len.brown@intel.com>,
        Chen Yu <yu.chen.surf@gmail.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 3/4] sched/fair: Calculate the scan depth for idle
 balance based on system utilization
Message-ID: <ZKwlomD5Xez7TRhy@chenyu5-mobl2>
References: <cover.1686554037.git.yu.c.chen@intel.com>
 <bc96b53efb712337a645e2c2a340975545ed5a28.1686554037.git.yu.c.chen@intel.com>
 <ZIqpI0ScCT5pmrAO@BLR-5CG11610CF.amd.com>
 <ZJKm195gy7X80wjm@chenyu5-mobl2.ccr.corp.intel.com>
 <7e31ad34-ce2c-f64b-a852-f88f8a5749a6@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <7e31ad34-ce2c-f64b-a852-f88f8a5749a6@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dXsfqb0FgGLweIfh32T74SkSHfSMDTkUGBwX/n2O1vaL+Mp0FgVjQ60lSx+7?=
 =?us-ascii?Q?DghfN0e7mTVckzzyIlZKbbbGhK42xLe/LaEJaJq4A/UMgvtmD7mmwoIlZwIs?=
 =?us-ascii?Q?Cj7e4WzE8zIV+7zSLLuB+dQcjZ/r91n6KJbnNNfdp5eftp09x5/ia9hE3ZPf?=
 =?us-ascii?Q?xZSLYM+Yj7Zt9ezr0zFBxonOqxFoVB1OHfgjxCIVXx+U1UYdKUlu5fmnCavB?=
 =?us-ascii?Q?+vjdOPuWhXlazsWiMpnroqXWuS2OYh54cpW0gO40sDFFvGc7SmF6KqgyVGZK?=
 =?us-ascii?Q?hp0x4Mvetlvg721R0SB8Tf/KBQ9pPIHydNJGNF0Z3EVzwTtmYbzlxhIgSNSq?=
 =?us-ascii?Q?jF3y0A6FBu8qsgbvfkxLYVTw2MufT1QiwGBPxlAUX7bvwUypTmSjRI3BmDUQ?=
 =?us-ascii?Q?HcJwUVaNayHfrzAX9wumnYvKGMRQdPj+Ir6xP1GIG8rHyu1s98bzEIkwL56i?=
 =?us-ascii?Q?nuuNPbJrNqBYzKmlnP2singgfFu+LwvvfaMceG/XvptUGFc1KN2kUSJUKqxD?=
 =?us-ascii?Q?j7dzjNI3/SPs4Jhw2pbsPpdUKjkgYt08nKwCiA4MwxAXwyREqHemwt8sNfTI?=
 =?us-ascii?Q?EC1QGt4tQ7Nt9q9Kzvy+xPji3AS5tZ6DdNUoy0ohoVP75FlY9Bmp5C2lfVBj?=
 =?us-ascii?Q?qAnPpKRJBye4wTJhYWePs1xKTFFB7SZFfFJ/cW9M+x+UM28kbh1y98BDny0n?=
 =?us-ascii?Q?9Q85GknbYQr7s1b623Q2BX8jEq4MDZxRo42vO8TJN39nqGCq1Cv0VfpLDTzd?=
 =?us-ascii?Q?xF7ypeCfiPSNRUhFkeyMVNOxpM1jx6NS2A5Cez2m1WWDUAswDKto3q3qwEXu?=
 =?us-ascii?Q?45RI+U42TUZLRWbVbkLsyIgucfwmAzHjCpSdu8l31KPoIt8pwd8VUi8DqqHx?=
 =?us-ascii?Q?rZBiit4mEdvjvOtQ8SduJIMHK5sFAaZig4MNG8DlqWiKF8bLnfXzpqj6Rr5p?=
 =?us-ascii?Q?Ae+0z02/fn8hr8Yxn5kWUjNIi18ZnJtzODhwEaUUvfTjYrw0h3tTcF6TlAy+?=
 =?us-ascii?Q?5e6hNOVjXZ06hQ96amGnYriWviEdtLLBEmVO418iBk0knz/L1qhPa6uGSNgr?=
 =?us-ascii?Q?eJ217ARSlqzAJa5sPv8XI7MUmda/AoQc8qjDXMpzGnBOIUJdm0QNsXiGFEE3?=
 =?us-ascii?Q?Y+KfcY12s+SDlnc/xTJ6c1eyAjzaP9SrWQwopl0aPNt6uxTZGWxDnmuE/bJ1?=
 =?us-ascii?Q?lObetUwXdagXhl2/clYGX0Se31s061qY8ZFick51UM6xxH09oNfkbZ5nf8d9?=
 =?us-ascii?Q?NRALl0AAiCQXKQzgDosLg325syyoiSSH7YN5iMBQ8Un72Dwzo1eiNSE37LlW?=
 =?us-ascii?Q?2FnfnTiU0gUFR0r7UXF9lMwIn3Lai8YpUitmQlneNjMUEVv8wAxXEnPiZIh6?=
 =?us-ascii?Q?b12XyGv/Ef4t4wikxCoJBf5Wd3XDeseadSJYzp9B4KYDbyfmrSx/MUNhGoYB?=
 =?us-ascii?Q?fioksTK4mxtcY6gKnVt7iTBliHcpfzZEy1APJeqUIsMeniQiDrcsIV2vcGlu?=
 =?us-ascii?Q?Tg6aq7T4XRBNN7eRgZ6JlE11mcel9ITP4kPaejzwMbaTm2c87g3LgrJN0Alq?=
 =?us-ascii?Q?9QoYH5z75jUwiwJREoL1W8cHuCr4kBDM4leDUhcZ?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c28ee099-931a-4bf0-eca3-08db815b8dbf
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jul 2023 15:37:22.6524
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2xCDs0N7Mt7KNDfvwBfiHiobQKr3fMQSpQCDwKpbvQRa+Z378sdkVlj4RtgBDna+QfQAvkMrezFROw1+ctNtuQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4761
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Prateek,

thanks for testing this patch,
On 2023-07-10 at 16:36:47 +0530, K Prateek Nayak wrote:
> Hello Chenyu,
> 
> Thank you for sharing this extended version. Sharing the results from
> testing below.
> 
> tl;dr
> 
> - tbench, netperf and unixbench-spawn see an improvement with ILB_UTIL.
> 
> - schbench (old) sees a regression in tail latency once system is heavily 
>   loaded. DeathStarBench and SPECjbb too see a small drop under those
>   conditions.
> 
> - Rest of the benchmark results do not vary much.
> 
>
[...] 
> I have a couple of theories:
> 
Thanks for the insights, I agree the risk you mentioned below could impact the
performance. Some thoughts below:
> o Either new_idle_balance is failing to find an overloaded busy rq as a
>   result of the limit.
> 
If the system is overloaded, the ilb_util finds a relatively busy rq and pulls from it.
There could be no much difference between a relatively busy rq and the busiest one,
because all rqs are quite busy I suppose?
> o Or, there is a chain reaction where pulling from a loaded rq which is not
>   the most loaded, will lead to more new_idle_balancing attempts which is
>   degrading performance.
Yeah, it could be possible that the ilb_util finds a relatively busy rq, but the
imbalance is not high so ilb decides not pull from it. However the busiest
rq is still waiting for someone to help, and this could trigger idle load
balance more frequently.
> 
> I'll go back and get some data to narrow down the cause. Meanwhile if
> there is any specific benchmark you would like me to run on the test
> system, please do let me know.
> 
Another hints might be that, as Gautham and Peter suggested, we should apply ILB_UTIL
to non-Numa domains. In above patch all the domains has sd_share which could
bring negative impact when accessing/writing cross-node data.
Sorry I did not post the latest version with Numa domain excluded previously as
I was trying to create a protype to further speed up idle load balance by
reusing the statistic suggested by Peter. I will send them out once it is stable.

Thanks again,
Chenyu
> --
> Thanks and Regards,
> Prateek