Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5ffb7ebcc75c106d791d4b5c4dece4b74c551245.camel@gmx.de>
Subject: Re: [RFC PATCH] sched/fair: Introduce SIS_PAIR to wakeup task on
 local idle core first
From:   Mike Galbraith <efault@gmx.de>
To:     Chen Yu <yu.c.chen@intel.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Tim Chen <tim.c.chen@intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Abel Wu <wuyun.abel@bytedance.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        Len Brown <len.brown@intel.com>,
        Chen Yu <yu.chen.surf@gmail.com>,
        Arjan Van De Ven <arjan.van.de.ven@intel.com>,
        Aaron Lu <aaron.lu@intel.com>, Barry Song <baohua@kernel.org>,
        linux-kernel@vger.kernel.org
Date:   Mon, 22 May 2023 09:10:33 +0200
In-Reply-To: <ZGrrUVZBY6qqeS0K@chenyu5-mobl1>
References: <20230516011159.4552-1-yu.c.chen@intel.com>
         <19664c68f77f5b23a86e5636a17ad2cbfa073f78.camel@gmx.de>
         <ZGNBt7vWJ3fDs5Sc@chenyu5-mobl1>
         <795a6d9475ecb444d219a9d36dc93b48a69e960e.camel@gmx.de>
         <ZGUHa+Si4dJbdsZN@chenyu5-mobl1>
         <0ac968e3-cd80-6339-970d-37005876b145@amd.com>
         <ZGWmveL2YTiXp2XR@chenyu5-mobl1>
         <12061140-4f09-b83f-843c-2fb8ff9f6e81@amd.com>
         <ZGrrUVZBY6qqeS0K@chenyu5-mobl1>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.42.4 
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
UI-OutboundReport: notjunk:1;M01:P0:DEeaW/FeMj8=;txeV46sf0K9oCdxYVmTyIKRbnxz
 AsP+jyK6QstlT3TgsrtaXoNuHW9QkgaL0YcDgxgdeNueFutqqppy+I7TB0MXwixcduC4VrFex
 Rloj0E/FHz8PH6oCKavzdRdEHOnBpLUkIAgBo1SzdiaxQ236tTmBDakMW2/f2DaedL937aV8L
 43E3ZxyPTyHWLIPg4SorbkZu/jN+s6gcw03ElbDni/LdTskG4KCU8B9Tyyc2Ml2ZVqsYprACY
 gby3Q9ksPBZPXDSo61ntmrQhHTf9vRb1QPSnNMozXXir4pvmAuUhsmyH9I0CGD3e9hRkqs8jZ
 KgTs1uWssmjrABeROl0h9UEVHJovEp6dAjxGHAVuAMhNsm2IOwbeZRwonP7oJh6dKubkLekIa
 Lr0AbxYtDE0+pkCSwKpbRCXjErps2oM/EuVp8CHenN/2YE+0Mr6nqoRv8WiVyPGEiQtz1gXEj
 oJk/xLxEXr+90ANKEJKKmwOr685ecNgUpfVA7QUGTV/d4XTKAmbOJsbAgb4QsA26SIqt3BTiB
 A4azS5cI9DI9pxq4xkepU782ggpY82DkOM23vBghx3aXK1o0KgwuR7/Rv/IYu/pa3gnuzciNE
 X/afTWMX1GNEwP5qAFRTFelX8gLzrgkLhGTmni1uHdSdPhDJv1EiIhCkFd9nJVnWtGPog+fLe
 MDl70c9Mp8vJhyjqUFaGXghXfzr0uYXIl7F1Pf5CSlfL/3TVuUYqXZFs3R0glL4K4+7wm7tFP
 ZxKaBkLTdRpLc8hSO6noYnKeEM30B+PJrexqQqNwoB8OPFmq9bja9X01nG19SyDbQGRRt613Q
 EeQpU9/ogP6G5e+aM8iqLXs2TJW/NKXJcXb2zcSFcZqirUx4s0whb1CdHYBgDkWH4e8hcW0QB
 WXtUQ/dXzwJFCTmKRr2UPnWrZyCE/U8E0ZlDB54dS+X1+dYjISqRQV3xujQZavKmKT+Sj8BHf
 L8zHqeQX0Je4Imyhc3sbwanOUno=
Precedence: bulk

On Mon, 2023-05-22 at 12:10 +0800, Chen Yu wrote:
>
> Meanwhile, I looked back at Yicong's proposal on waking up task
> on local cluster first. It did show some improvement on Jacobsville,
> I guess that could also be a chance to reduce C2C latency.

Something else to consider is that communication data comes in many
size chunks and volumes, and cache footprints can easily be entirely
local constructs unrelated to any transferred data.

At one extreme of the huge spectrum of possibilities, a couple less
than brilliant tasks playing high speed ping-pong can bounce all over a
box with zero consequences, but for a pair more akin to say Einstein
and Bohr pondering chalkboards full of mind bending math and meeting
occasionally at the water cooler to exchange snarky remarks, needlessly
bouncing them about forces them to repopulate chalkboards, and C2C
traffic you try to avoid via bounce you generate via bounce.

Try as you may, methinks you're unlikely to succeed at avoiding C2C in
a box where roughly half of all paths are C2C.  What tasks have in
their pockets and what they'll do with a CPU at any point in time is
unknown and unknowable by the scheduler, dooming pinpoint placement
accuracy as a goal.

	-Mike