Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Date:   Sat, 7 Oct 2023 11:23:06 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     K Prateek Nayak <kprateek.nayak@amd.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Vincent Guittot" <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>, Aaron Lu <aaron.lu@intel.com>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        "Daniel Bristot de Oliveira" <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        <linux-kernel@vger.kernel.org>, Chen Yu <yu.chen.surf@gmail.com>
Subject: Re: [PATCH 0/2] Introduce SIS_CACHE to choose previous CPU during
 task wakeup
Message-ID: <ZSDPGjO+hkD0AjJ/@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1695704179.git.yu.c.chen@intel.com>
 <27651e14-f441-c1e2-9b5b-b958d6aadc79@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <27651e14-f441-c1e2-9b5b-b958d6aadc79@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/LQyGmyzk7NAk3d6tk/n+On0U2yBsIbwkWRR+8F27AIyGjZjzbrgt6yCmVh7?=
 =?us-ascii?Q?NUkCzZcm/5GGiaBIwnBeZFJta1lYA5tr4yMC8zwyLNUazBNnvnEKbqyiKOqQ?=
 =?us-ascii?Q?gpzSRzAwGdKnpL00BIV3hEuzMrrbEr4qy5KONqia55fxDrfUOmJG9SyPLCLB?=
 =?us-ascii?Q?46d26YsA0VV1L49qbf1+AtjiqEf0emb2QlTTRMWoQKWw05BXQJkBcJzaS195?=
 =?us-ascii?Q?sR4uRRYNY4AOs1KHR/OnAkLuCHryhWE2yhKO7KtEm56OjUWQ2K0pFCUiE71R?=
 =?us-ascii?Q?VVbEBtb/CGXsPEy9wqafDuC9fyJTT/z915CLn85xSjBrwbP5F8sdhBn1xq3q?=
 =?us-ascii?Q?GYtZlTSVkzfn3vQLnfU2VVBdGm2IEEfRMN7nB5HfkZIQxtDfjY7BzrjDqefl?=
 =?us-ascii?Q?qEJCDmOjyV63zS3Dn7iHreyFGUKO662Nk4GarhzkNt2sydt2ePKqvDlL+6Iz?=
 =?us-ascii?Q?G5EGchHB5tmeyy0lk55qJFCm85w+3jYGoUtMUYC5vc3jUX8AqZRfcidmBPcE?=
 =?us-ascii?Q?lp2pq9I4t+fgLTGnuS8UN/OZaZ9CSXY060p6pPRn01ruVF19dupMUobn0D1u?=
 =?us-ascii?Q?9yChw+V391bVn5c9u2KymM3ol3Jro3MCYRKLyJ3fRsUDISDn0UCLi3aMTpuv?=
 =?us-ascii?Q?JyoL1pesol7mQf1w9mYLIPPx0FTWgl967OHprgqEuzE7sBp0+4zoXJ+tpGWw?=
 =?us-ascii?Q?Jdg80lci8/L+Zj1rYC/T/o9JqUVyl2SoDX5WbCfopJ0LVoRL7Uzp8J3kdr9W?=
 =?us-ascii?Q?jRA6rDwPMDF/riSIFxzFzvbVqbGITwqHdxGVe6U/TwlP/QqT8i9Gr6197w39?=
 =?us-ascii?Q?1JL0QXp0G9eLWMpDOGxkw5KrJ+tj5QRLQSrrXmclG/TZs2CCKFN6Uicslfrr?=
 =?us-ascii?Q?LZJdRNI/KFdP81jJJmJ3j6K+cnNCZ23IEzQwKMhB9tGy0d07L9jYgm4RCYE9?=
 =?us-ascii?Q?Plm1K/y2RXSiANzEjNZfh4p/U8S6oxKLBt0N9Ri6NAH/2VWIR66ODotcfSP1?=
 =?us-ascii?Q?XO/DNh03+8bnph8XEggGIUtMRcyPOnLSIOvEHyU0v4DFr/dTvWMIoM8ZR13b?=
 =?us-ascii?Q?AlQZcXP8FnWM36EP0FksIymArFFYJV/18lWrHX3kWYPze86uImyJh9sqJOn5?=
 =?us-ascii?Q?9PrrBYpS4V7K97/dDXW3TY7O7mYMsdvRvpTVhYnx0finMC+aXR8QVG2WWAqx?=
 =?us-ascii?Q?Sz55HT6QlTsmbI14XqMMPRS0yxym294m53RDScrWsIEIvXpSQ/Lbs95bYwPL?=
 =?us-ascii?Q?/eOda/wN00XI/it6s6VTmjyeUwOzn7sFHjDrzNXaX6k45vQVPl2t7oXq9398?=
 =?us-ascii?Q?fR5BHkAP8xv7fWQJWPknxCbO4uThT3RKi+Q9+lLXbvi3qAjr5z8csCVOxURR?=
 =?us-ascii?Q?rsFbgYBfFMST3+44f+HJVzz2k1hB9xDmCypQJ1izrETOhxasR1hHBUxjogrj?=
 =?us-ascii?Q?Qgrne6aiVDAOyet1D3ov4p4Fb0HIXc35Dk2eIaHuMH6l1lN+vYQZA9oyHAxr?=
 =?us-ascii?Q?Arv4td5YAr7AK3ZaJPKfaKVtH849M/XHkrHZHR7K3Do/Q7kWMsriZf6AGFgp?=
 =?us-ascii?Q?2GlQIAveG4/eHvB/juwr2gQBBe4pjKfFv26MhPNe?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d3c6b54b-58f2-4a0d-23aa-08dbc6e4c414
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2023 03:23:25.3014
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UVEDMHf08Oco/a216T4nuLJ2A4TyY5YcklJGFtd4ehe/2+dMuk39XR2VAIBJISQ2v+nJN79st/xUXaYc6ksi6w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7963
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 06 Oct 2023 20:23:45 -0700 (PDT)
X-Spam-Level: **

Hi Prateek,

On 2023-10-05 at 11:52:13 +0530, K Prateek Nayak wrote:
> Hello Chenyu,
> 
> On 9/26/2023 10:40 AM, Chen Yu wrote:
> > RFC -> v1:
> > - drop RFC
> > - Only record the short sleeping time for each task, to better honor the
> >   burst sleeping tasks. (Mathieu Desnoyers)
> > - Keep the forward movement monotonic for runqueue's cache-hot timeout value.
> >   (Mathieu Desnoyers, Aaron Lu)
> > - Introduce a new helper function cache_hot_cpu() that considers
> >   rq->cache_hot_timeout. (Aaron Lu)
> > - Add analysis of why inhibiting task migration could bring better throughput
> >   for some benchmarks. (Gautham R. Shenoy)
> > - Choose the first cache-hot CPU, if all idle CPUs are cache-hot in
> >   select_idle_cpu(). To avoid possible task stacking on the waker's CPU.
> >   (K Prateek Nayak)
> > 
> > Thanks for your comments and review!
> 
> Sorry for the delay! I'll leave the test results from a 3rd Generation
> EPYC system below.
> 
> tl;dr
> 
> - Small regression in tbench and netperf possible due to more searching
>   for an idle CPU.
> 
> - Small regression in schbench (old) at 256 workers albeit with large
>   run to run variance.
> 
> - Other benchmarks are more or less same.
> 
> Test          : schbench
> Units         : Normalized 99th percentile latency in us
> Interpretation: Lower is better
> Statistic     : Median
> ==================================================================
> #workers:  tip[pct imp](CV)     SIS_CACHE[pct imp](CV)
>   1     1.00 [ -0.00]( 3.95)     0.97 [  2.56](10.42)
>   2     1.00 [ -0.00]( 5.89)     0.83 [ 16.67](22.56)
>   4     1.00 [ -0.00](14.28)     1.00 [ -0.00](14.75)
>   8     1.00 [ -0.00]( 4.90)     0.84 [ 15.69]( 6.01)
>  16     1.00 [ -0.00]( 4.15)     1.00 [ -0.00]( 4.41)
>  32     1.00 [ -0.00]( 5.10)     1.01 [ -1.10]( 3.44)
>  64     1.00 [ -0.00]( 2.69)     1.04 [ -3.72]( 2.57)
> 128     1.00 [ -0.00]( 2.63)     0.94 [  6.29]( 2.55)
> 256     1.00 [ -0.00](26.75)     1.51 [-50.57](11.40)

Thanks for the testing. So the latency regression from schbench is
quite obvious, and as you mentioned, it is possible due to longer
scan time during select_idle_cpu(). I'll run the same test with split
LLC to see if I can reproduce the issue or not.
I'm also working with Mathieu on another direction to choose previous CPU
over current CPU when the system is overloaded, and that should be
more moderate and I'll post the test result later.

thanks,
Chenyu