Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sun, 7 May 2023 10:19:38 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Dietmar Eggemann <dietmar.eggemann@arm.com>
CC:     Ingo Molnar <mingo@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Qais Yousef <qyousef@layalina.io>,
        Kajetan Puchalski <kajetan.puchalski@arm.com>,
        "Morten Rasmussen" <morten.rasmussen@arm.com>,
        Vincent Donnefort <vdonnefort@google.com>,
        Quentin Perret <qperret@google.com>,
        "Abhijeet Dharmapurikar" <adharmap@quicinc.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/1] sched: Consider CPU contention in frequency &
 load-balance busiest CPU selection
Message-ID: <ZFcKus/4Jo5oRuU8@chenyu5-mobl1>
References: <20230406155030.1989554-1-dietmar.eggemann@arm.com>
 <20230406155030.1989554-2-dietmar.eggemann@arm.com>
 <ZFSr4Adtx1ZI8hoc@chenyu5-mobl1>
 <8276e5ce-b1f6-19e2-095b-abc6952bd1c7@arm.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <8276e5ce-b1f6-19e2-095b-abc6952bd1c7@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UnMl9VBRjZ/pEMw+uhbXoAUkvDLHf+ap55BoKQoDomgAPrJfml99X7fuA6UB?=
 =?us-ascii?Q?qEUEtdLYXskTAJulanYwyofyE4Pr0j0H4vVa7brGX43TZ0FQdWYeMIb3H4ft?=
 =?us-ascii?Q?OjZamda/+FhE/zbQlabcG6HsZv37SoApSdvL8ozQloqtrPuD9uIV9+PaZXGl?=
 =?us-ascii?Q?Xsr8/wAqQyKkal6gqTYNYK6+TPsyJmzNdo7tdsVcDmJZxqX+VOXmaPtNpyyL?=
 =?us-ascii?Q?FqrZ15yZI+ovNCqm8M6rQVRgVheulvumtllLKdGCNNX5AolqHZT/ApaeE520?=
 =?us-ascii?Q?zchGttoGZat1r2HhSXmTE2BFgmq44kaKy3hGFl2xvxXlEJbDWfp9ZQX62Vq8?=
 =?us-ascii?Q?13YMgCCxCrJyayTEvkyAawrWDZ3GUsvc0rzfeCkSvdnwGUFyWuAwDBW+zelC?=
 =?us-ascii?Q?jRafajGogYS/qf1lg8F8x2+lwY0JNtDrfXowlb+HflSG5vEn2ktKo/Ym9MOY?=
 =?us-ascii?Q?KRfBC0vuGMTS9C+9zXgc0+StrJsWWqqNQwcnoXbvwYPipq7wdIp+5MVp5r6O?=
 =?us-ascii?Q?VpKvmt1gHTov5bkckECq8HgHCeWsB+0fw3+gyZOdpYH78Nf8WDjXbPoChZK0?=
 =?us-ascii?Q?7ORpR+6ngpqax2qE2BabcJdDx+HY5aDNci0Y2eDWRS1yOtrNvssnvaKySnWD?=
 =?us-ascii?Q?I0UkTDZxoWPLNV/12nh65RuMEhpcTkNZMFsOGMhXx2401nVAAwXY7G0tIUYa?=
 =?us-ascii?Q?yrQfI2Jwjxd8v57Vu3BnQrmU8zPsdGrj75TvN4LWJ3rD3olpuqoXI7wo9gCf?=
 =?us-ascii?Q?EN3iJj+SjhQKnAHq3O/WXA1r1Bey3VjSRqmLUOnKv5nNbbMLSSgPbCxQidkk?=
 =?us-ascii?Q?LuErZ+p2DLjL8oq6+To/qw+bHscyg7hro1WL7k3715kOx7a2JLc6aRGRX9Py?=
 =?us-ascii?Q?qv0PFN7uMjlU9KPNWxAlMSqbcy4C4BIA6gAkrU4Z50zJj/AHVm+P7itXYgdh?=
 =?us-ascii?Q?7fo/H1AH+LXhD6bfpasYMOYEBv1l+9ks66ND/Iz9NaGxKVWBUhBc3N1uLmWV?=
 =?us-ascii?Q?nAbjeyNbi9c/lI/jNXLBiAKiwu+s919h9YwTh7/UIVjjcskhfbWDTaGT4i2e?=
 =?us-ascii?Q?eHhnbQTABBSdXbVVAeceMsuUzIYXoJgptUlEe1JVBP2qBja65EpKYZymhcdW?=
 =?us-ascii?Q?tZqe9OnusOKagperGuG/K5cdMZUZHbrpa3h7FZCG0PzhiIvlP5kuuW9+aR7W?=
 =?us-ascii?Q?+IshytDsPHSc30QW9cHBWR11nq2dzLjRlmQpTRsETYAEe0VGB3MSf7SnPuRE?=
 =?us-ascii?Q?7oMcqI52UEbtm30FpUZitRMe+G91wPrwOYQw8k6TUczEGb6Aa3UDhlFP+xOw?=
 =?us-ascii?Q?SQZgK8+qgPcHio0AX/1rkCGKDDeYt3iiCBS+PJG63e3rWMbiI3nWmgPbP6l3?=
 =?us-ascii?Q?6yZiGhMj9lFgiLKykaxXmqkG91cWM47/2+8KZvSWDkp003ti4KKcL0oJxQVc?=
 =?us-ascii?Q?IsjDlPYDR/RP5Yemx0GKtAAjacq+GiIWj4SetZo6WAItKAUBjAxJE6uxv8Lm?=
 =?us-ascii?Q?nBSOsk5gQMZdnzWoTghcOCCIvcVmO1VWQhrrEGk+P12AeAMsYUwSuQnURFag?=
 =?us-ascii?Q?dWpZrtrFtfB3cYbhAGnic35MBIoeleUUVT/rpz3x?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 56b9c90a-d4ea-45da-ead7-08db4ea18bc1
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2023 02:19:54.7854
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XMLRPDaovjFpqWUoPYhM3+Cvj45hLaU/AISsux+WkN6Hg1tN5kDM0Y79KT4vGKDChD2GbjdyBu9LKolEAtlGKw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7638
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2023-05-05 at 20:02:36 +0200, Dietmar Eggemann wrote:
> Hi Chenyu,
> 
> On 05/05/2023 09:10, Chen Yu wrote:
> > On 2023-04-06 at 17:50:30 +0200, Dietmar Eggemann wrote:
> >> Use new cpu_boosted_util_cfs() instead of cpu_util_cfs().
> >>
> >> The former returns max(util_avg, runnable_avg) capped by max CPU
> >> capacity. CPU contention is thereby considered through runnable_avg.
> >>
> >> The change in load-balance only affects migration type `migrate_util`.
> >>
> >> Suggested-by: Vincent Guittot <vincent.guittot@linaro.org>
> >> Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
> >>
> > Tested on Intel Sapphire Rapids which has 2x56C/112T = 224 CPUs.
> > The test tries to check if this is any impact on find_busiest_queue()
> > so it was tested with cpufreq governor performance.
> > The baseline is the 6.3 sched/core branch on top of
> > Commit 67fff302fc445a ("sched/fair: Introduce SIS_CURRENT to wake up"),
> > and compared to the code with current patch applied.
> > 
> > In summary no obvious difference and some small improvements on tbench
> > were observed so far:
> 
> many thanks for the test results!
> 
> Could you share the parameter lists you use for the individual tests?
> This would make it easier to understand the results and rerun the tests
> on similar machines.
>
Yes. The test parameters are all tested with cpufreq governor performance,
turbo disabled and C-states > C1 are disabled.

netperf:
Loopback test. Launched 1 netserver. Launched 25%, 50%, 75%, 100%, 125%, 150%, 175%, 200%
of the online CPUs number of netperf client. Tested type is TCP_RR and UDP_RR
respectively.

for i in $(seq 1 $nr_netperf); do
    netperf -4 -H 127.0.0.1 -t $type -c -C -l 100 &
done

tbench:
Loopback test. Launched 1 tbench_srv. Launched 25%, 50%, 75%, 100%, 125%, 150%, 175%, 200%
of the online CPUs number of tbench client.

tbench -t 100 $nr_tbench 127.0.0.1


schbench:
Number of worker threads is 25% of the online CPUs. Number of message threads is
1, 2, 4, 8 respectively.

schbench -m $nr_message -t $nr_worker -r 100


hackbench:
Number of hackbench group is 1, 2, 4, 8 respectively. Work types are "process" and "threads"
respectively. IPC mode are "pipe" and "sockets". The number of fd in each group is 12.5% of
the CPU number.

hackbench -g $nr_group --$work_type --$ipc_mode -l 5000000 -s 100 -f $num_fds


thanks,
Chenyu