Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 28 Apr 2021 18:19:25 +0530
From:   Srikar Dronamraju <srikar@linux.vnet.ibm.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>
Cc:     Ingo Molnar <mingo@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Mel Gorman <mgorman@techsingularity.net>,
        Rik van Riel <riel@surriel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Gautham R Shenoy <ego@linux.vnet.ibm.com>,
        Parth Shah <parth@linux.ibm.com>
Subject: Re: [PATCH 00/10] sched/fair: wake_affine improvements
Message-ID: <20210428124925.GM2633526@linux.vnet.ibm.com>
Reply-To: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
References: <20210422102326.35889-1-srikar@linux.vnet.ibm.com>
 <CAKfTPtAuFpr05-ZBNjB9OiNNQnmgPSX3S4=Sz-A8sOnFAkr7Tg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <CAKfTPtAuFpr05-ZBNjB9OiNNQnmgPSX3S4=Sz-A8sOnFAkr7Tg@mail.gmail.com>
Precedence: bulk

* Vincent Guittot <vincent.guittot@linaro.org> [2021-04-27 16:52:30]:

> Hi Srikar,

Hi Vincent, 

> On Thu, 22 Apr 2021 at 12:23, Srikar Dronamraju
> <srikar@linux.vnet.ibm.com> wrote:
> >
> > Recently we found that some of the benchmark numbers on Power10 were lesser
> > than expected. Some analysis showed that the problem lies in the fact that
> > L2-Cache on Power10 is at core level i.e only 4 threads share the L2-cache.
> >
> >
> > Summary:
> > mc-llc outperforms, this patchset and upstream almost give similar performance.
> 
> So mc-llc patch seems to be the best approach IMHO. Although the
> hemisphere don't share cache, they share enough resources so
> cache-snooping is as efficient as sharing cache
> 

Yes, mc-llc helps just specific systems like Power10 but its shows better
numbers than my posted patchset.

However in this patchset, we are looking at areas in wakeup (aka idler llcs)
we could optimize which can help other archs too. + the fallback mechanism
is generic enough that we could use it for other Systems too.

I know that there are valid concerns raised by Mel and I working to resolve
them. Some of them are.
- How hot is idle-core
- Crashes when running tbench (I was able to reproduce with kernbench on x86)

Also I am adding some more changes with which we are getting similar
performance as mc-llc.

> > Cc: LKML <linux-kernel@vger.kernel.org>
> > Cc: Michael Ellerman <mpe@ellerman.id.au>
> > Cc: Gautham R Shenoy <ego@linux.vnet.ibm.com>
> > Cc: Parth Shah <parth@linux.ibm.com>
> > Cc: Ingo Molnar <mingo@kernel.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Valentin Schneider <valentin.schneider@arm.com>
> > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> > Cc: Mel Gorman <mgorman@techsingularity.net>
> > Cc: Vincent Guittot <vincent.guittot@linaro.org>
> > Cc: Rik van Riel <riel@surriel.com>
> >
> > Srikar Dronamraju (10):
> >   sched/fair: Update affine statistics when needed
> >   sched/fair: Maintain the identity of idle-core
> >   sched/fair: Update idle-core more often
> >   sched/fair: Prefer idle CPU to cache affinity
> >   sched/fair: Call wake_affine only if necessary
> >   sched/idle: Move busy_cpu accounting to idle callback
> >   sched/fair: Remove ifdefs in waker_affine_idler_llc
> >   sched/fair: Dont iterate if no idle CPUs
> >   sched/topology: Introduce fallback LLC
> >   powerpc/smp: Add fallback flag to powerpc MC domain
> >
> >  arch/powerpc/kernel/smp.c      |   7 +-
> >  include/linux/sched/sd_flags.h |   7 +
> >  include/linux/sched/topology.h |   3 +-
> >  kernel/sched/fair.c            | 229 +++++++++++++++++++++++++++------
> >  kernel/sched/features.h        |   1 +
> >  kernel/sched/idle.c            |  33 ++++-
> >  kernel/sched/sched.h           |   6 +
> >  kernel/sched/topology.c        |  54 +++++++-
> >  8 files changed, 296 insertions(+), 44 deletions(-)
> >
> > --
> > 2.18.2
> >

-- 
Thanks and Regards
Srikar Dronamraju