Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Date:   Tue, 12 Sep 2023 22:51:57 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Mike Galbraith <efault@gmx.de>
CC:     K Prateek Nayak <kprateek.nayak@amd.com>,
        Tim Chen <tim.c.chen@intel.com>, Aaron Lu <aaron.lu@intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        "Mel Gorman" <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        <linux-kernel@vger.kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>
Subject: Re: [RFC PATCH 2/2] sched/fair: skip the cache hot CPU in
 select_idle_cpu()
Message-ID: <ZQB7DWSuUmzql8/D@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1694397335.git.yu.c.chen@intel.com>
 <d49cf5748aa7c6d69580315d2373a9eafa21c21f.1694397335.git.yu.c.chen@intel.com>
 <31c910c2-717d-b86c-4e08-5c94383682e8@amd.com>
 <ZP7ptt70uF10wxlg@chenyu5-mobl2>
 <a8c1b296a4ed444b9cab166a5a39aa11836903a8.camel@gmx.de>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <a8c1b296a4ed444b9cab166a5a39aa11836903a8.camel@gmx.de>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?onyEsKueWA8o3Xpd6+bBM8KBnmGYBwpFOyT3eSFEki4vHlpxRM1Pwy2GxCW7?=
 =?us-ascii?Q?LlzRTrjKjWLvSxxIZ4WwCiakv/txn3AWxxCd+4YfFUa9gYsyJMtvLRucn8rS?=
 =?us-ascii?Q?Rv+GKgsC2OinM7REjyS63SAtkGB9zMZFNIHXHefOpC/96+T6CEK5p5Vb4W5l?=
 =?us-ascii?Q?JP6ik7SkyY90RxGmZXgADtgd4EZBuWeDzjEKRJUfuAAGJyroRMLsfwHrkbzM?=
 =?us-ascii?Q?KL63UXgVnAy67NZgXPXoj6bhIgyDL/aoGn+Rt8VpPhWMXdvAEPq3dbFFp2CI?=
 =?us-ascii?Q?FNxSM/KK+6morQpS8gzzttf9OYZiaw61Sbn27ko/6cmGVDUBOwYbULu20DSg?=
 =?us-ascii?Q?PC5qDZWDODIpaEDb8I1EHMjTpoZdIbcgHW4nn0HyLk/FzY5iyAQH2wo2N9dx?=
 =?us-ascii?Q?uRzuZVomsGorq7VHl91eB/VcNDuGPSLDwIEDAMywlncE6eXGfoMF0kpMGSGs?=
 =?us-ascii?Q?eqcxTPLbgw3rha+3yBGwKcsAPjTGwYWg/uEDuWeUjt7XvYgO/XqVHYegB7fq?=
 =?us-ascii?Q?2Kt7tItbmzwte8Jy20Qpju0Gq1nUL5XY/IEK0X9wRUhq8phD1eqDyL67vr3u?=
 =?us-ascii?Q?a23ISAeB2ozFLJ0YAa9gUXZ7a7tr4STcsRA+vWgPJS/lTKqwqa6elNIGO6Aw?=
 =?us-ascii?Q?Pt9fBz1gfN0MwJW3Ts+lN38ImnQMzA7TQDmMqUV9oxyFPKXdNdRWtmYmrKsU?=
 =?us-ascii?Q?JZpMW8xFdl+SDH25n3kOmPkwJJIRPwpfdMm6ceR2eqTsEVExd8d3DzJ7tDI6?=
 =?us-ascii?Q?67tF/g71TFzOTaJzaxlE4pgmiyjAK05cBmYZJNt1eOEJnAXMpfqWqOVNMl+O?=
 =?us-ascii?Q?GmCIp/M7o/noNUWN30itlaBEtkxueLbhndnbSNEVWirqtni5F466JDH5CV5g?=
 =?us-ascii?Q?MhbeiFCBeuj4XbjeZAuLNXTzNZl+zn/JV/0148KHYH++vtoEVkejaVJSO7VI?=
 =?us-ascii?Q?n4VMgrDh11cwE/ikTOcbDEztlz8ZLlGrwWVG1V527bblEc/UvY5v6CvhHDYx?=
 =?us-ascii?Q?3hO052Ew7WJMx/mDluYidyWNlUrM8fCSYmP7UgCaMhJyT2az4gj38OyCPJoI?=
 =?us-ascii?Q?0FTXnaWuNAVGSjFzYZHep+CD6glKpGI8cAgmHa1i36fAL0sqRr66hNHLpzXo?=
 =?us-ascii?Q?gFm7T5yJFdx1Gq2Tw6eit/BFlXuOY2UN/8iyzYFJTbQR3srqdKmxq0H0FD0E?=
 =?us-ascii?Q?0rK7291TIZGoaK26Lw1tfk83f7XWpJzdg4vLy9LneSlgFPftMHZeeHCbyOQO?=
 =?us-ascii?Q?poPLTVti5HgEBEH3HbRKajHp0n/NeTV7mQCf4rDXxYtRGHImkbL+mnx1BmRh?=
 =?us-ascii?Q?+KDbwgpnsYexbaNt60hx9twlSbpAETNauKPqkhmabov6PS6NrNveTMxdh4G5?=
 =?us-ascii?Q?Fsmdtd7gQHz16/7wej6tcOfzJsmn9ImpfOO4bavLpKqYdzS2PA34m1vDo0NA?=
 =?us-ascii?Q?8BJpXpPzaw+dLY+favn8MhxvxvJbs0dJGPpPyYBHYkvqHgfTHmQbLVCf5BSM?=
 =?us-ascii?Q?EMgglS951Mjluz8LeHCqTaQWVw5QMFKDF6iEK6GRwmDw5favRNgMerMd5Tn4?=
 =?us-ascii?Q?7q8zptmGLiUCHY8K+qxLrelAVmd7yHnlXnK1Ej/K?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c1294e81-cc30-4a50-568a-08dbb39fda8c
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2023 14:52:15.3009
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ymZ8loOrUbq2GvWhLljQ9jhHwRdmEdNnikoXLXIXWQKbfIortLJd+QeYbTvpflFohq9FskhU8rfj3JfW0DP4LQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8361
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 12 Sep 2023 07:52:22 -0700 (PDT)

Hi Mike,

thanks for taking a look,

On 2023-09-12 at 11:39:55 +0200, Mike Galbraith wrote:
> On Mon, 2023-09-11 at 18:19 +0800, Chen Yu wrote:
> >
> > > Speaking of cache-hot idle CPU, is netperf actually more happy with
> > > piling on current CPU?
> >
> > Yes. Per my previous test, netperf of TCP_RR/UDP_RR really likes to
> > put the waker and wakee together.
> 
> Hm, seems there's at least one shared L2 case where that's untrue by
> more than a tiny margin, which surprised me rather a lot.
> 

Yes, the task stacking is in theory against the work conservation of the
scheduler, and it depends on how much the resource(l1/l2 cache, dsb) locallity
is, and it is workload and hardware specific.

> For grins, I tested netperf on my dinky rpi4b, and while its RR numbers
> seem kinda odd, they're also seemingly repeatable (ergo showing them).
> I measured a very modest cross-core win on a shared L2 Intel CPU some
> years ago (when Q6600 was shiny/new) but nothing close to these deltas.
> 

This is interesting, I have a Jacobsville which also has shared L2, I'll
run some tests to check what the difference between task stacking vs spreading task
on that platform. But I guess that is another topic because current patch
avoids stacking tasks.

thanks,
Chenyu

> Makes me wonder what (a tad beefier) Bulldog RR numbers look like.
> 
> root@rpi4:~# ONLY=TCP_RR netperf.sh
> TCP_RR-1        unbound    Avg:  29611  Sum:    29611
> TCP_RR-1        stacked    Avg:  22540  Sum:    22540
> TCP_RR-1        cross-core Avg:  30181  Sum:    30181
> 
> root@rpi4:~# netperf.sh
> TCP_SENDFILE-1  unbound    Avg:  15572  Sum:    15572
> TCP_SENDFILE-1  stacked    Avg:  11533  Sum:    11533
> TCP_SENDFILE-1  cross-core Avg:  15751  Sum:    15751
> 
> TCP_STREAM-1    unbound    Avg:   6331  Sum:     6331
> TCP_STREAM-1    stacked    Avg:   6031  Sum:     6031
> TCP_STREAM-1    cross-core Avg:   6211  Sum:     6211
> 
> TCP_MAERTS-1    unbound    Avg:   6306  Sum:     6306
> TCP_MAERTS-1    stacked    Avg:   6094  Sum:     6094
> TCP_MAERTS-1    cross-core Avg:   9393  Sum:     9393
> 
> UDP_STREAM-1    unbound    Avg:  22277  Sum:    22277
> UDP_STREAM-1    stacked    Avg:  18844  Sum:    18844
> UDP_STREAM-1    cross-core Avg:  24749  Sum:    24749
> 
> TCP_RR-1        unbound    Avg:  29674  Sum:    29674
> TCP_RR-1        stacked    Avg:  22267  Sum:    22267
> TCP_RR-1        cross-core Avg:  30237  Sum:    30237
> 
> UDP_RR-1        unbound    Avg:  36189  Sum:    36189
> UDP_RR-1        stacked    Avg:  27129  Sum:    27129
> UDP_RR-1        cross-core Avg:  37033  Sum:    37033