Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <88e9287ef7a86f24999af00f90d6f122de024979.camel@linux.intel.com>
Subject: Re: [PATCH v2 2/2] sched/fair: Scan cluster before scanning LLC in
 wake-up path
From:   Tim Chen <tim.c.chen@linux.intel.com>
To:     Barry Song <21cnbao@gmail.com>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc:     "Gautham R. Shenoy" <gautham.shenoy@amd.com>,
        Yicong Yang <yangyicong@hisilicon.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        LKML <linux-kernel@vger.kernel.org>,
        LAK <linux-arm-kernel@lists.infradead.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        prime.zeng@huawei.com,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        ego@linux.vnet.ibm.com, Linuxarm <linuxarm@huawei.com>,
        Barry Song <song.bao.hua@hisilicon.com>,
        Guodong Xu <guodong.xu@linaro.org>, yu.c.chen@intel.com
Date:   Fri, 04 Feb 2022 09:41:21 -0800
In-Reply-To: <CAGsJ_4zVEjFYoff=x=Y3i9xPxoi891x-gkfA6Lsdc+yT2ykRmQ@mail.gmail.com>
References: <20220126080947.4529-1-yangyicong@hisilicon.com>
         <20220126080947.4529-3-yangyicong@hisilicon.com>
         <YfK9DSMFabjYm/MV@BLR-5CG11610CF.amd.com>
         <CAGsJ_4xL3tynB9P=rKMoX2otW4bMMU5Z-P9zSudMV3+fr2hpXw@mail.gmail.com>
         <20220128071337.GC618915@linux.vnet.ibm.com>
         <CAGsJ_4yoUONACY-j+9XxSNC0VgmdyRdHC=z87dWvZvVSASzXRQ@mail.gmail.com>
         <20220201093859.GE618915@linux.vnet.ibm.com>
         <CAGsJ_4z8cer7Y5si+J_=awQetFJZMVeaQ+RDSXQz9EGOPTGMQg@mail.gmail.com>
         <20220204073317.GG618915@linux.vnet.ibm.com>
         <CAGsJ_4xjgy3D0VzbTdmJihJ+nut_NeTEb4krh8jup4rbvTY_ww@mail.gmail.com>
         <CAGsJ_4zVEjFYoff=x=Y3i9xPxoi891x-gkfA6Lsdc+yT2ykRmQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) 
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Fri, 2022-02-04 at 23:49 +1300, Barry Song wrote:
> On Fri, Feb 4, 2022 at 11:28 PM Barry Song <21cnbao@gmail.com> wrote:
> > On Fri, Feb 4, 2022 at 8:33 PM Srikar Dronamraju
> > <srikar@linux.vnet.ibm.com> wrote:
> > > * Barry Song <21cnbao@gmail.com> [2022-02-02 09:20:32]:
> > > 
> > > > On Tue, Feb 1, 2022 at 10:39 PM Srikar Dronamraju
> > > > <srikar@linux.vnet.ibm.com> wrote:
> > > > > * Barry Song <21cnbao@gmail.com> [2022-01-28 07:40:15]:
> > > > > 
> > > > > > On Fri, Jan 28, 2022 at 8:13 PM Srikar Dronamraju
> > > > > > <srikar@linux.vnet.ibm.com> wrote:
> > > > > > > * Barry Song <21cnbao@gmail.com> [2022-01-28 09:21:08]:
> > > > > > > 
> > > > > > > > On Fri, Jan 28, 2022 at 4:41 AM Gautham R. Shenoy
> > > > > > > > <gautham.shenoy@amd.com> wrote:
> > > > > > > > > On Wed, Jan 26, 2022 at 04:09:47PM +0800, Yicong Yang
> > > > > > > > > wrote:
> > > > > > > > > > From: Barry Song <song.bao.hua@hisilicon.com>
> > > > > > > > > > 
> > > > > > I am sorry I didn't get your question. Currently the code
> > > > > > works as below:
> > > > > > if task A wakes up task B, and task A is in LLC0 and task B
> > > > > > is in LLC1.
> > > > > > we will scan the cluster of A before scanning the whole
> > > > > > LLC0, in this case,
> > > > > > cluster of A is the closest sibling, so it is the better
> > > > > > choice than other CPUs
> > > > > > which are in LLC0 but not in the cluster of A.
> > > > > 
> > > > > Yes, this is right.
> > > > > 
> > > > > > But we do scan all cpus of LLC0
> > > > > > afterwards if we fail to find an idle CPU in the cluster.
> > > > > 
> > > > > However my reading of the patch, before we can scan other
> > > > > clusters within
> > > > > the LLC (aka LLC0), we have a check in scan cluster which
> > > > > says
> > > > > 
> > > > >         /* Don't ping-pong tasks in and out cluster
> > > > > frequently */
> > > > >         if (cpus_share_resources(target, prev_cpu))
> > > > >            return target;
> > > > > 
> > > > > My reading of this is, ignore other clusters (at this point,
> > > > > we know there
> > > > > are no idle CPUs in this cluster. We don't know if there are
> > > > > idle cpus in
> > > > > them or not) if the previous CPU and target CPU happen to be
> > > > > from the same
> > > > > cluster. This effectively means we are given preference to
> > > > > cache over idle
> > > > > CPU.
> > > > 
> > > > Note we only ignore other cluster while prev_cpu and target are
> > > > in same
> > > > cluster. if the condition is false, we are not ignoring other
> > > > cpus. typically,
> > > > if waker is the target, and wakee is the prev_cpu, that means
> > > > if they are
> > > > already in one cluster, we don't stupidly spread them in
> > > > select_idle_cpu() path
> > > > as benchmark shows we are losing. so, yes, we are giving
> > > > preference to
> > > > cache over CPU.
> > > 
> > > We already figured out that there are no idle CPUs in this
> > > cluster. So dont
> > > we gain performance by picking a idle CPU/core in the
> > > neighbouring cluster.
> > > If there are no idle CPU/core in the neighbouring cluster, then
> > > it does make
> > > sense to fallback on the current cluster.
> > 
> > 

We may need to take into consideration the utilization and
load average for the source and target cluster to make
better decision of whether it is worth placing the
task in the next cluster.  If the load of the target
cluster is too high, it is not worth pushing the task there.

Those stats can be gathered during load balancing without adding
overhead in the hot task wakeup path.

Chen Yu played around with cutting off the idle CPU search
in a LLC based on such stats and he saw some good
improvements over the default.

Tim