Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 30 Oct 2019 13:44:40 -0400
From:   Phil Auld <pauld@redhat.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>
Cc:     Valentin Schneider <valentin.schneider@arm.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Ingo Molnar <mingo@kernel.org>,
        Mel Gorman <mgorman@techsingularity.net>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        Quentin Perret <quentin.perret@arm.com>,
        Morten Rasmussen <Morten.Rasmussen@arm.com>,
        Hillf Danton <hdanton@sina.com>,
        Parth Shah <parth@linux.ibm.com>,
        Rik van Riel <riel@surriel.com>
Subject: Re: [PATCH v4 00/10] sched/fair: rework the CFS load balance
Message-ID: <20191030174440.GI1686@pauld.bos.csb>
References: <20191024123844.GB2708@pauld.bos.csb>
 <20191024134650.GD2708@pauld.bos.csb>
 <CAKfTPtB0VruWXq+wGgvNOMFJvvZQiZyi2AgBoJP3Uaeduu2Lqg@mail.gmail.com>
 <20191025133325.GA2421@pauld.bos.csb>
 <CAKfTPtDWV7AkzMNuJtkN-pLmDcK41LwNiX0Wr8UT+vMFHAx6Qg@mail.gmail.com>
 <20191030143937.GC1686@pauld.bos.csb>
 <564ca629-5c34-dbd1-8e64-2da6910b18a3@arm.com>
 <bf96be8a-2358-b9ab-b8eb-d0b8b94ed0d7@arm.com>
 <20191030171914.GF1686@pauld.bos.csb>
 <CAKfTPtDVJH_eGiHCyz1Boz4m0tqMP3rgbSoudZ+9kPXB4_aGnQ@mail.gmail.com>
MIME-Version: 1.0
In-Reply-To: <CAKfTPtDVJH_eGiHCyz1Boz4m0tqMP3rgbSoudZ+9kPXB4_aGnQ@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Oct 30, 2019 at 06:28:50PM +0100 Vincent Guittot wrote:
> On Wed, 30 Oct 2019 at 18:19, Phil Auld <pauld@redhat.com> wrote:
> >
> > Hi,
> >
> > On Wed, Oct 30, 2019 at 05:35:55PM +0100 Valentin Schneider wrote:
> > >
> > >
> > > On 30/10/2019 17:24, Dietmar Eggemann wrote:
> > > > On 30.10.19 15:39, Phil Auld wrote:
> > > >> Hi Vincent,
> > > >>
> > > >> On Mon, Oct 28, 2019 at 02:03:15PM +0100 Vincent Guittot wrote:
> > > >
> > > > [...]
> > > >
> > > >>>> When you say slow versus fast wakeup paths what do you mean? I'm=
 still
> > > >>>> learning my way around all this code.
> > > >>>
> > > >>> When task wakes up, we can decide to
> > > >>> - speedup the wakeup and shorten the list of cpus and compare onl=
y
> > > >>> prev_cpu vs this_cpu (in fact the group of cpu that share their
> > > >>> respective LLC). That's the fast wakeup path that is used most of=
 the
> > > >>> time during a wakeup
> > > >>> - or start to find the idlest CPU of the system and scan all doma=
ins.
> > > >>> That's the slow path that is used for new tasks or when a task wa=
kes
> > > >>> up a lot of other tasks at the same time
> > > >
> > > > [...]
> > > >
> > > > Is the latter related to wake_wide()? If yes, is the SD_BALANCE_WAK=
E
> > > > flag set on the sched domains on your machines? IMHO, otherwise tho=
se
> > > > wakeups are not forced into the slowpath (if (unlikely(sd))?
> > > >
> > > > I had this discussion the other day with Valentin S. on #sched and =
we
> > > > were not sure how SD_BALANCE_WAKE is set on sched domains on
> > > > !SD_ASYM_CPUCAPACITY systems.
> > > >
> > >
> > > Well from the code nobody but us (asymmetric capacity systems) set
> > > SD_BALANCE_WAKE. I was however curious if there were some folks who s=
et it
> > > with out of tree code for some reason.
> > >
> > > As Dietmar said, not having SD_BALANCE_WAKE means you'll never go thr=
ough
> > > the slow path on wakeups, because there is no domain with SD_BALANCE_=
WAKE for
> > > the domain loop to find. Depending on your topology you most likely w=
ill
> > > go through it on fork or exec though.
> > >
> > > IOW wake_wide() is not really widening the wakeup scan on wakeups usi=
ng
> > > mainline topology code (disregarding asymmetric capacity systems), wh=
ich
> > > sounds a bit... off.
> >
> > Thanks. It's not currently set. I'll set it and re-run to see if it mak=
es
> > a difference.
>=20
> Because the fix only touches the slow path and according to Valentin
> and Dietmar comments on the wake up path, it would mean that your UC
> creates regularly some new threads during the test ?
>=20

I believe it is not creating any new threads during each run.=20


> >
> >
> > However, I'm not sure why it would be making a difference for only the =
cgroup
> > case. If this is causing issues I'd expect it to effect both runs.
> >
> > In general I think these threads want to wake up the last cpu they were=
 on.
> > And given there are fewer cpu bound tasks that CPUs that wake cpu shoul=
d,
> > more often than not, be idle.
> >
> >
> > Cheers,
> > Phil
> >
> >
> >
> > --
> >

--=20