Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <5eba2fb4af9ebc7396101bb9bd6c8aa9c8af0710.1571899508.git.viresh.kumar@linaro.org>
 <20191030164714.GH28938@suse.de> <20191108113156.y555y4se2mshv7in@vireshk-i7>
In-Reply-To: <20191108113156.y555y4se2mshv7in@vireshk-i7>
From:   Vincent Guittot <vincent.guittot@linaro.org>
Date:   Fri, 8 Nov 2019 18:01:19 +0100
Message-ID: <CAKfTPtDraLuk7Cu0scLQBdcPh2CaK8CdSuGUzYFNACNEao-mvQ@mail.gmail.com>
Subject: Re: [PATCH] sched/fair: Make sched-idle cpu selection consistent throughout
To:     Viresh Kumar <viresh.kumar@linaro.org>
Cc:     Mel Gorman <mgorman@suse.de>, Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        linux-kernel <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, 8 Nov 2019 at 12:32, Viresh Kumar <viresh.kumar@linaro.org> wrote:
>
> On 30-10-19, 16:47, Mel Gorman wrote:
> > On Thu, Oct 24, 2019 at 12:15:27PM +0530, Viresh Kumar wrote:
> > > There are instances where we keep searching for an idle CPU despite
> > > having a sched-idle cpu already (in find_idlest_group_cpu(),
> > > select_idle_smt() and select_idle_cpu() and then there are places where
> > > we don't necessarily do that and return a sched-idle cpu as soon as we
> > > find one (in select_idle_sibling()). This looks a bit inconsistent and
> > > it may be worth having the same policy everywhere.
> > >
> >
> > This needs supporting data.
>
> I did some more interesting tests with rt-app. It was getting
> difficult to generate the correct numbers with normal use cases as
> most of the time prev/target/etc CPUs were found to be completely idle
> and the task was getting placed there in all the cases and so no diff
> with sched-idle changes.
>
> To prove the point I was making (that we can reduce task latency with
> SCHED_IDLE), I created 3 different tests on my hikey board (octa-core,
> 2 clusters, 0-3 and 4-7). The cpufreq governor was set to performance
> to avoid any side affects from CPU frequency.
>
> Test 1: 1-cfs-task:
>
> A single SCHED_NORMAL task is pinned to CPU5 which runs for 2333 us
> out of 7777 us (so gives time for the cluster to go in deep idle
> state).
>
> Test 2: 1-cfs-1-idle-task:
>
> A single SCHED_NORMAL task is pinned on CPU5 and single SCHED_IDLE
> task is pinned on CPU6 (to make sure cluster 1 doesn't go in deep idle
> state).
>
> Test 3: 1-cfs-8-idle-task:
>
> A single SCHED_NORMAL task is pinned on CPU5 and eight SCHED_IDLE
> tasks are created which run forever (not pinned anywhere, so they run
> on all CPUs). Checked with kernelshark that as soon as NORMAL task
> sleeps, the SCHED_IDLE task starts running on CPU5.
>
> And here are the results on mean latency (in us), using the "st" tool.
>
> $ st 1-cfs-task/rt-app-cfs_thread-0.log
> N       min     max     sum     mean    stddev
> 642     90      592     197180  307.134 109.906
>
> $ st 1-cfs-1-idle-task/rt-app-cfs_thread-0.log
> N       min     max     sum     mean    stddev
> 642     67      311     113850  177.336 41.4251
>
> $ st 1-cfs-8-idle-task/rt-app-cfs_thread-0.log
> N       min     max     sum     mean    stddev
> 643     29      173     41364   64.3297 13.2344
>
>
> The mean latency when:
> - we need to wakeup from deep idle state is 307 us
> - we need to wakeup from shallow idle state is 177 us
> - we need to preempt a SCHED_IDLE task is 64 us
>
> So the theory looks correct, we should probably prefer SCHED_IDLE CPUs
> both for power and performance :)
>
> > find_idlest_group_cpu is generally from
> > a fork() context where it's not particularly performance critical.
> > select_idle_sibling and the helpers it uses is wakeup context where is
> > is often much more critical to wake quickly than find the best CPU.
>
> I agree. We must find the best CPU here. But won't a SCHED_IDLE cpu be
> the best ? After all that is the one in shallowest idle state and so
> better for power :)

It makes sense to me to consider a CPU that runs only SCHED_IDLE task
as an idle CPU with shortest latency and most recently idled
timestamp. This seems to be confirmed be the data above.
The SCHED_IDLE tasks would be somewhat penalized because they can now
be preempted whereas there is a real idle CPU but such  SCHED_IDLE
task don't have any other requirements than not delaying NORMAL task
wakeup
Also this simplifies and shortens the search loop.


>
> --
> viresh