Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Date:   Thu, 19 Oct 2023 18:57:00 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Madadi Vineeth Reddy <vineethr@linux.ibm.com>
CC:     <cover.1695704179.git.yu.c.chen@intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>, Aaron Lu <aaron.lu@intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        "Steven Rostedt" <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        "Valentin Schneider" <vschneid@redhat.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        <linux-kernel@vger.kernel.org>, Chen Yu <yu.chen.surf@gmail.com>,
        "Madadi Vineeth Reddy" <vineethr@linux.ibm.com>
Subject: Re: [PATCH 0/2] Introduce SIS_CACHE to choose previous CPU during
 task wakeup
Message-ID: <ZTELfNyUDAL0s36C@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1695704179.git.yu.c.chen@intel.com>
 <3f98806b-fd74-cfba-b48c-2526109d10a3@linux.ibm.com>
 <ZS5rhO5XysGOUn4M@chenyu5-mobl2.ccr.corp.intel.com>
 <eb45778d-3302-2ece-8d2e-319b1fcd071d@linux.ibm.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <eb45778d-3302-2ece-8d2e-319b1fcd071d@linux.ibm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?DNfyXpA75sGEdm67/LtcBEvSmQPeKWBu8tbwdvrLlwVRk/II5IUCsigefpf2?=
 =?us-ascii?Q?C48ojzW+aT9tFaSv3Nd1A9SvRGOuRR/6XTVJoByGzGNDOe5hrEQ+hOw7HGVi?=
 =?us-ascii?Q?v9ksauGdBgFHprI/bcvgQiz8vGDRYC2N0HWS4MEUB9qa/z+viChEaBwvSo+O?=
 =?us-ascii?Q?UzpRrLuo0bgKh8pHsU2nXgTD/KwomZXai+30G5UH7JAUBGsoXyL820X13xUX?=
 =?us-ascii?Q?YfcJ1Gkgb+NTJoEKjCCVBCz2WDiyjmruET7gLQzD8JU2Kle0L7Ql0trqleZ+?=
 =?us-ascii?Q?qeq4ru6jGVqvPpjg7j7NXfU/a9czyq5D5LseRwPVdsL1HvxSLM0WMvir1Pf4?=
 =?us-ascii?Q?oh+Tdp53Ouo8Kfcp7VLmgzOsULrpY5dAdVyrS0ELWyDAz9lW5A7ATCoy2gq7?=
 =?us-ascii?Q?Xp9EEWiOpl6CY/5RfaofTiION/eQUP5x2gKHG4opORe66dShHMdc5NtmbGnY?=
 =?us-ascii?Q?AA11hRI+zbMKlkmN6AmegwCbZBkTAvwCV5YmuLgsgQKvjjto6MVZ2aP5nFpS?=
 =?us-ascii?Q?F0l9VS29IA6DELt4HmSyyhIWIpc8UOkTfVBB6sTKuUgdL8eKIQA6/RzKb6UI?=
 =?us-ascii?Q?jSr60LQ7bpGEPmN/7i9nxWpU71yoHfAgHQ8RSjd5sjvfb3f5B7K2pGmMu6DN?=
 =?us-ascii?Q?5fW9hxpIODlu7kmIceOOBftZOT2bYMi+mykDsOZkkAdoGVibywxW67+kVUi5?=
 =?us-ascii?Q?lWp3+vw7mlsk10u5lHBsGkibqK3+4gNbxRsZ6gDbF9Oh3888mhh4Uv8flKlw?=
 =?us-ascii?Q?b09VjoRvPyCLyvNT+UUsTtE04hwVmub7UOx47lpeCTnm7VI3cAop1sjpfWfl?=
 =?us-ascii?Q?sKFaVCgt+dSil748RkS5QhdK76AQ8nSadh9ee4xjtbgyZWZkALSqz2X7WyXb?=
 =?us-ascii?Q?wrzrx0ACh6LsqxLhokveNUkn89lRQnxAhxWxTPXHxjpGY8Uw/MGbUT0lxtk5?=
 =?us-ascii?Q?kk6kwiSEqYM53YZj3c6uyu19YBctdewYLJZHni5qs1kLtJZVYw1yJjQiM5P+?=
 =?us-ascii?Q?21kFxeka1pv8ywBBVe11SJ5L3Hyp2FlHq0xpH789nHp43t6r7qyz8AEVYqbR?=
 =?us-ascii?Q?zSHqHbCB8TlJ6KmhrvAUSp4IsqXr6UE+hB0Xg43ky2WhE9ceAkFSfW5TebiD?=
 =?us-ascii?Q?WExIE9GYUrn3zY2++dLykiokwWSdUJLDK3mPufg6EwEb+Q1htOt8Wt54aEJC?=
 =?us-ascii?Q?dwC1RHaU4Mwd5iDbCrqtFkmUUxzKI3fKiAzpQq7W5o7hjr02yU/b6v/hVzU0?=
 =?us-ascii?Q?kt2ggI0zlDdwfgS7IWW44EnovYUTZI8BtXS4vmpIGj3IxpdsNScxE+n6kihT?=
 =?us-ascii?Q?6aSlBe16iS+AdSgQExXIpkSWW6iX9PKL20RhMnbviM//n/1OGTQOtQ9gcU2B?=
 =?us-ascii?Q?HDUv/d+nfYszgpoGY20FaPQKTFR3lBfMEBM8PgZ+B+w7j2o3+lS9aZwJ0vle?=
 =?us-ascii?Q?BwxGMx6CzUJvEDP3kwEETlXLUJyrPpRCYmUSaniCmKV3yAtv+QY2cyrXm6wc?=
 =?us-ascii?Q?BOh5F55JXdlewAzdCiNwEzxCmU6G2xMtQ/hUyTwv+Fm5p0kYy3CbfSvChs8K?=
 =?us-ascii?Q?lIgxuGY2DMtmyW05Csb46ePaqqCrHZS0k5lG60KE?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5ef1da7f-2e85-428e-b1d6-08dbd0922768
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2023 10:57:15.2122
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MJfAJqXxduwuHDATeGEL7vrQjunb9g/tnDjtz29+xnSUU954KfGi3VXEmBz0AKupDAeB918R8YIX47co91J9nA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6917
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 19 Oct 2023 03:58:00 -0700 (PDT)

On 2023-10-19 at 01:02:16 +0530, Madadi Vineeth Reddy wrote:
> Hi Chen Yu,
> On 17/10/23 16:39, Chen Yu wrote:
> > Hi Madadi,
> > 
> > On 2023-10-17 at 15:19:24 +0530, Madadi Vineeth Reddy wrote:
> >> Hi Chen Yu,
> >>
> >> On 26/09/23 10:40, Chen Yu wrote:
> >>> RFC -> v1:
> >>> - drop RFC
> >>> - Only record the short sleeping time for each task, to better honor the
> >>>   burst sleeping tasks. (Mathieu Desnoyers)
> >>> - Keep the forward movement monotonic for runqueue's cache-hot timeout value.
> >>>   (Mathieu Desnoyers, Aaron Lu)
> >>> - Introduce a new helper function cache_hot_cpu() that considers
> >>>   rq->cache_hot_timeout. (Aaron Lu)
> >>> - Add analysis of why inhibiting task migration could bring better throughput
> >>>   for some benchmarks. (Gautham R. Shenoy)
> >>> - Choose the first cache-hot CPU, if all idle CPUs are cache-hot in
> >>>   select_idle_cpu(). To avoid possible task stacking on the waker's CPU.
> >>>   (K Prateek Nayak)
> >>>
> >>> Thanks for your comments and review!
> >>>
> >>> ----------------------------------------------------------------------
> >>
> >> Regarding making the scan for finding an idle cpu longer vs cache benefits, 
> >> I ran some benchmarks.
> >>
> > 
> > Thanks very much for your interest and your time on the patch.
> > 
> >> Tested the patch on power system with 12 cores. Total of 96 CPU's.
> >> System has two NUMA nodes.
> >>
> >> Below are some of the benchmark results
> >>
> >> schbench 99.0th latency (lower is better)
> >> ========
> >> case            load        	baseline[pct imp](std%)       SIS_CACHE[pct imp]( std%)
> >> normal          1-mthreads      1.00 [ 0.00]( 3.66)            1.00 [  0.00]( 1.71)
> >> normal          2-mthreads      1.00 [ 0.00]( 4.55)            1.02 [ -2.00]( 3.00)
> >> normal          4-mthreads      1.00 [ 0.00]( 4.77)            0.96 [ +4.00]( 4.27)
> >> normal          6-mthreads      1.00 [ 0.00]( 60.37)           2.66 [ -166.00]( 23.67)
> >>
> >>
> >> schbench results are showing that there is not much impact in wakeup latencies due to more iterations 
> >> in search for an idle cpu in the select_idle_cpu code path and interestingly numbers are slightly better 
> >> for SIS_CACHE in case of 4-mthreads.
> > 
> > The 4% improvement is within std%, so I suppose we did not see much difference in 4 mthreads case.
> > 
> >> I think we can ignore the last case due to huge run to run variations.
> > 
> > Although the run-to-run variation is large, it seems that the decrease is within that range.
> > Prateek has also reported that when the system is overloaded there could be some regression
> > from schbench:
> > https://lore.kernel.org/lkml/27651e14-f441-c1e2-9b5b-b958d6aadc79@amd.com/
> > Could you also post the raw data printed by schbench? And maybe using the latest schbench could get the
> > latency in detail.
> >  
> 
> raw data by schbench(old) with 6-mthreads
> ======================
> 
> Baseline (5 runs)
> ========
> Latency percentiles (usec)                                                                                                                                                                                                                                  
>         50.0000th: 22
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 37
>         *99.0000th: 981 
>         99.5000th: 4424
>         99.9000th: 9200
>         min=0, max=29497
> 
> Latency percentiles (usec)
>         50.0000th: 23
>         75.0000th: 29
>         90.0000th: 35
>         95.0000th: 38
>         *99.0000th: 495 
>         99.5000th: 3924
>         99.9000th: 9872
>         min=0, max=29997
> 
> Latency percentiles (usec)
>         50.0000th: 23
>         75.0000th: 30
>         90.0000th: 36
>         95.0000th: 39
>         *99.0000th: 1326
>         99.5000th: 4744
>         99.9000th: 10000
>         min=0, max=23394
> 
> Latency percentiles (usec)
>         50.0000th: 23
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 37
>         *99.0000th: 55
>         99.5000th: 3292
>         99.9000th: 9104
>         min=0, max=25196
> 
> Latency percentiles (usec)
>         50.0000th: 23
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 37
>         *99.0000th: 711 
>         99.5000th: 4600
>         99.9000th: 9424
>         min=0, max=19997
> 
> SIS_CACHE (5 runs)
> =========
> Latency percentiles (usec)                                                                                                                                                                                                                                                                                     
>         50.0000th: 23
>         75.0000th: 30
>         90.0000th: 35
>         95.0000th: 38
>         *99.0000th: 1894
>         99.5000th: 5464
>         99.9000th: 10000
>         min=0, max=19157
> 
> Latency percentiles (usec)
>         50.0000th: 22
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 37
>         *99.0000th: 2396
>         99.5000th: 6664
>         99.9000th: 10000
>         min=0, max=24029
> 
> Latency percentiles (usec)
>         50.0000th: 22
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 37
>         *99.0000th: 2132
>         99.5000th: 6296
>         99.9000th: 10000
>         min=0, max=25313
> 
> Latency percentiles (usec)
>         50.0000th: 22
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 37
>         *99.0000th: 1090
>         99.5000th: 6232
>         99.9000th: 9744
>         min=0, max=27264
> 
> Latency percentiles (usec)
>         50.0000th: 22
>         75.0000th: 29
>         90.0000th: 34
>         95.0000th: 38
>         *99.0000th: 1786
>         99.5000th: 5240
>         99.9000th: 9968
>         min=0, max=24754
> 
> The above data as indicated has large run to run variation and in general, the latency is
> high in case of SIS_CACHE for the 99th %ile.
> 
> 
> schbench(new) with 6-mthreads
> =============
> 
> Baseline
> ========
> Wakeup Latencies percentiles (usec) runtime 30 (s) (209403 total samples)
> 	  50.0th: 8          (43672 samples)
> 	  90.0th: 13         (83908 samples)
> 	* 99.0th: 20         (18323 samples)
> 	  99.9th: 775        (1785 samples)
> 	  min=1, max=8400
> Request Latencies percentiles (usec) runtime 30 (s) (209543 total samples)
> 	  50.0th: 13648      (59873 samples)
> 	  90.0th: 14000      (82767 samples)
> 	* 99.0th: 14320      (16342 samples)
> 	  99.9th: 18720      (1670 samples)
> 	  min=5130, max=38334
> RPS percentiles (requests) runtime 30 (s) (31 total samples)
> 	  20.0th: 6968       (8 samples)
> 	* 50.0th: 6984       (23 samples)
> 	  90.0th: 6984       (0 samples)
> 	  min=6835, max=6991
> average rps: 6984.77
> 
> 
> SIS_CACHE
> =========
> Wakeup Latencies percentiles (usec) runtime 30 (s) (209295 total samples)
> 	  50.0th: 9          (49267 samples)
> 	  90.0th: 14         (86522 samples)
> 	* 99.0th: 21         (14091 samples)
> 	  99.9th: 1146       (1722 samples)
> 	  min=1, max=10427
> Request Latencies percentiles (usec) runtime 30 (s) (209432 total samples)
> 	  50.0th: 13616      (62838 samples)
> 	  90.0th: 14000      (85301 samples)
> 	* 99.0th: 14352      (16149 samples)
> 	  99.9th: 21408      (1660 samples)
> 	  min=5070, max=41866
> RPS percentiles (requests) runtime 30 (s) (31 total samples)
> 	  20.0th: 6968       (7 samples)
> 	* 50.0th: 6984       (21 samples)
> 	  90.0th: 6984       (0 samples)
> 	  min=6672, max=6996
> average rps: 6981.07
> 
> In new schbench, I didn't observe run to run variation and also there was no regression
> in case of SIS_CACHE for the 99th %ile.
>

Thanks for the test Madadi, in my opinion we can stick with the new schbench
in the future. I'll have a double check on my test machine.

thanks,
Chenyu