Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 25 May 2023 15:47:36 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Mike Galbraith <efault@gmx.de>
CC:     K Prateek Nayak <kprateek.nayak@amd.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        "Ingo Molnar" <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Tim Chen <tim.c.chen@intel.com>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Abel Wu <wuyun.abel@bytedance.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        Len Brown <len.brown@intel.com>,
        Chen Yu <yu.chen.surf@gmail.com>,
        Arjan Van De Ven <arjan.van.de.ven@intel.com>,
        Aaron Lu <aaron.lu@intel.com>, Barry Song <baohua@kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Introduce SIS_PAIR to wakeup task on
 local idle core first
Message-ID: <ZG8SmByO0g5fjOc7@chenyu5-mobl1>
References: <20230516011159.4552-1-yu.c.chen@intel.com>
 <19664c68f77f5b23a86e5636a17ad2cbfa073f78.camel@gmx.de>
 <ZGNBt7vWJ3fDs5Sc@chenyu5-mobl1>
 <795a6d9475ecb444d219a9d36dc93b48a69e960e.camel@gmx.de>
 <ZGUHa+Si4dJbdsZN@chenyu5-mobl1>
 <0ac968e3-cd80-6339-970d-37005876b145@amd.com>
 <ZGWmveL2YTiXp2XR@chenyu5-mobl1>
 <12061140-4f09-b83f-843c-2fb8ff9f6e81@amd.com>
 <ZGrrUVZBY6qqeS0K@chenyu5-mobl1>
 <5ffb7ebcc75c106d791d4b5c4dece4b74c551245.camel@gmx.de>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <5ffb7ebcc75c106d791d4b5c4dece4b74c551245.camel@gmx.de>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NdSCfmMnnIaq42ku/rJYstdqcZ9RpNzcPw/p+lgEBTDlAfWKixACdL8ehkIK?=
 =?us-ascii?Q?hfLpCVg6njvKMEAUf4oSysokBG73WPQNetrdOQHkkQkQW+5yIx9eO9ahge1a?=
 =?us-ascii?Q?yJGjdl6HO2ikbdl+Qf/nxaPIgn7LwxR23D59c7KGvBgNwL0/zGE9QtKGrW0R?=
 =?us-ascii?Q?iV8BOG9hMDsEHlrXkfs+BhlZTkTyDmVGdl2FigrB0L5eo2K5/qW0aICoUc4e?=
 =?us-ascii?Q?ldJN+mmg0wuhwraGQAUrsI1nnbxReEZKJ4W1f8SFBToafwHynkBcptz/oBH5?=
 =?us-ascii?Q?TwKXxDyo4DJ43quYwPbWRjRqWFhfRykDYs15lfn9AmWVxLiAQqmVLhGQubJ6?=
 =?us-ascii?Q?B5PbaqT0O0LwaarnJeuNj+jTkL4IDL/ME+lzeCb3zPE/e9qY4MZCFozIXMRF?=
 =?us-ascii?Q?CrUfkBOIMimVC5uvnjr2wm11w4QM7DBWxII3tlWriD2atCswLhOym9ScH+at?=
 =?us-ascii?Q?bRth6q1pZ4vgb5fIxrT8n1joMgTXUP7LdMoWD+NRPOtGx1l0NEwUTYAGm1xJ?=
 =?us-ascii?Q?D2dQWVR1VsOpmmlBTVGHDELdJuKzxemU3efMUzCak4wq3WHo9lT7DDRjsiAq?=
 =?us-ascii?Q?wg/ufOxrimQaK8o4Chb85Y86Odwp6Ca/51M6yom+vCDZKoq04y50XLsSF95X?=
 =?us-ascii?Q?PlEwcG3O6DlHpq7p2prRR21Ey/Yy8G09RzKduURQ9cFLinRvE9oYb/27yWVy?=
 =?us-ascii?Q?oOwIkk+6IYs2cs08LTUrx79mtAf3YJA38+B4Uxz8AZigveTo0N1i3o0ls57s?=
 =?us-ascii?Q?GjgW1+59cTqy9GB1wqUXzgQnGsNPnQaxgpINj5dqllUOzhKUoaAlC0+iEz+o?=
 =?us-ascii?Q?ZJu8BgzCdQZ7FfwcZXfkNJFLU03WpT3IN/vgqtt1qCLFa3O1u7iO/ajCmrIX?=
 =?us-ascii?Q?tvpdxbrWu/2pyKRi3z5hMpDyD+jWHevuk7qQXPDNgSnyjJu1ZKEiB0N730qE?=
 =?us-ascii?Q?J+R3KTyKPKAIdk8IeEMQda9jW3Mnuxqyj3zYUDSKLiU0iK1M4hZdn4QGG0wS?=
 =?us-ascii?Q?qMhvgBlUWD89uvL9h+fLiLc5qXBIHRk0CMz53WsgGMdXskip/t4pL7nCv98H?=
 =?us-ascii?Q?uHvQv2or6kQO78f+wiYwy0tsU3czpo8fU6KqTi7X7Dmg2gXXKJKHzEqHTJOE?=
 =?us-ascii?Q?jJy93sM/gaM8nt2fgIBnq2SEXaZEiksZb6dP6sv95oQ6h5b743JVq1+r0h0F?=
 =?us-ascii?Q?R4EpkLzR0/MT3594ON1OG5sZ7V9m5hQi/8yjr7sbtatnr2a+dU9E2G9B4kNy?=
 =?us-ascii?Q?8r7pZnuBUM6NgZOfIBv7vWPzBLAaIWIxrivTwovevO1bzl7MSHxmnmdnJVSC?=
 =?us-ascii?Q?a7DXDfbc2U3vt7tpNQdI+PeB0v5+4eEMTbGh+nNTbH4+yTkBTGN4hvcy1Sz5?=
 =?us-ascii?Q?GQi9mLUfoLJqjFF+NhLEgR7r5FwPJ87ayKzjUHPIZ86+lYeZxQsfcl9NBS8T?=
 =?us-ascii?Q?5ZjHHFglw4h3KjXzjRkw5xcjXMTbP6Kp0zC/F+fVYllAAurMl0Jx8PbiFMwB?=
 =?us-ascii?Q?tW/HDDHV3ufa1db8+WJyOiQkkZY/v9oLtHUckmXseYujH4i/DqYwJIZoDcOW?=
 =?us-ascii?Q?2obCyn6H0QQYlqSI8qa7fe6uixj8ANpKdYHy7Tn/?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 76e0e9ae-105e-4e7d-38b2-08db5cf45a27
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 May 2023 07:47:55.9933
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LWalJ1K9Xjc+0tAhdilgpHlfzGrkne6T7sIKl3gAiivgRARo1P/xDqPkMGyIgh66ovu7HhQLJq1hrRvqYdrlOA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR11MB6508
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_MSPIKE_H2,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2023-05-22 at 09:10:33 +0200, Mike Galbraith wrote:
> On Mon, 2023-05-22 at 12:10 +0800, Chen Yu wrote:
> >
> > Meanwhile, I looked back at Yicong's proposal on waking up task
> > on local cluster first. It did show some improvement on Jacobsville,
> > I guess that could also be a chance to reduce C2C latency.
> 
> Something else to consider is that communication data comes in many
> size chunks and volumes, and cache footprints can easily be entirely
> local constructs unrelated to any transferred data.
> 
> At one extreme of the huge spectrum of possibilities, a couple less
> than brilliant tasks playing high speed ping-pong can bounce all over a
> box with zero consequences, but for a pair more akin to say Einstein
> and Bohr pondering chalkboards full of mind bending math and meeting
> occasionally at the water cooler to exchange snarky remarks, needlessly
> bouncing them about forces them to repopulate chalkboards, and C2C
> traffic you try to avoid via bounce you generate via bounce.
> 
> Try as you may, methinks you're unlikely to succeed at avoiding C2C in
> a box where roughly half of all paths are C2C.  What tasks have in
> their pockets and what they'll do with a CPU at any point in time is
> unknown and unknowable by the scheduler, dooming pinpoint placement
> accuracy as a goal.
>
I guess what you mean is that, for a wakee has large local data cache
footprint, it is not a good idea to wakeup the wakee on a remote core.
Because in that way the wakee has to repopulate the cache from scratch.
Yes, the problem is that currently the scheduler is lacking of metric
to indicate the task's working set, or per-task-cache-footprint-track
(although we have numa balancing to calculate per-task-node-statistics).
If provided with this cache-aware metric, the wakee can be put to a candidate
CPU where the cache locallity(either LLC or L2) is friendly to the wakee.
Because there is no such accurate metric, the heuristic seems to be an compromised
way to predict the task placement.

The C2C was mainly caused by accessing global tg->load, so besides
wakeup placement, there should also be other way to mitigate C2C,
such as reducing the frequency of accessing tg->load.

Besides that, while studying the history of wake_wide(), I suddenly
found that 10 years ago Michael has proposed exactly the same strategy to
check if task A and B are waking up each other, if they are, put them
together, otherwise, spread them to different LLC:
https://lkml.org/lkml/2013/3/6/73
And this version has finnaly evolved to what wake_wide() looks like today
in your patch:
https://marc.info/?l=linux-kernel&m=143688840122477
If I understand correctly, if wake_wide() can decide whether to wakeup the
task on an idle CPU on local LLC or remote LLC, does it also
make sense to extend wake_wide() for SMT domain and L2 domain?
Say, if wake_wide(p, nr_smt) returns true, then find an idle CPU on remote
SMT domain, otherwise scan for an idle CPU in local SMT domain. In this
case, does has_idle_core check matter?

thanks,
Chenyu