Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <DB4481A8-FD4E-4879-9CD2-275ABAFC09CF@fb.com>
In-Reply-To: <DB4481A8-FD4E-4879-9CD2-275ABAFC09CF@fb.com>
From:   Vincent Guittot <vincent.guittot@linaro.org>
Date:   Mon, 26 Oct 2020 09:39:32 +0100
Message-ID: <CAKfTPtBiOFXwV9SkZ=YBw16xoS6LSrKVR4sFX6r2hZPZ9_5-+A@mail.gmail.com>
Subject: Re: [PATCH] fix scheduler regression from "sched/fair: Rework load_balance()"
To:     Chris Mason <clm@fb.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Rik van Riel <riel@surriel.com>,
        linux-kernel <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi Chris

On Sat, 24 Oct 2020 at 01:49, Chris Mason <clm@fb.com> wrote:
>
> Hi everyone,
>
> We=E2=80=99re validating a new kernel in the fleet, and compared with v5.=
2,

Which version are you using ?
several improvements have been added since v5.5 and the rework of load_bala=
nce

> performance is ~2-3% lower for some of our workloads.  After some
> digging, Johannes found that our involuntary context switch rate was ~2x
> higher, and we were leaving a CPU idle a higher percentage of the time,
> even though the workload was trying to saturate the system.
>
> We were able to reproduce the problem with schbench, and Johannes
> bisected down to:
>
> commit 0b0695f2b34a4afa3f6e9aa1ff0e5336d8dad912
> Author: Vincent Guittot <vincent.guittot@linaro.org>
> Date:   Fri Oct 18 15:26:31 2019 +0200
>
>      sched/fair: Rework load_balance()
>
> Our working theory is the load balancing changes are leaving processes
> behind busy CPUs instead of moving them onto idle ones.  I made a few
> schbench modifications to make this easier to demonstrate:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/mason/schbench.git/
>
> My VM has 40 cpus (20 cores, 2 threads per core), and my schbench
> command line is:

What is the topology ? are they all part of the same LLC ?

>
> schbench -t 20 -r 0 -c 1000000 -s 1000 -i 30 -z 120
>
> This has two message threads, and 20 workers per message thread.  Once
> woken up, the workers think for a full second, which means you=E2=80=99ll=
 have
> some long latencies if you=E2=80=99re stuck behind one of these workers i=
n the
> runqueue.  The message thread does a little bit of work and then sleeps,
> so we end up with 40 threads hammering full blast on the CPU and 2
> threads popping in and out of idle.
>
> schbench times the delay from when a message thread wakes a worker to
> when the worker runs.  On a good kernel, the output looks like this:
>
> Latency percentiles (usec) runtime 1290 (s) (3280 total samples)
>          50.0th: 155 (1653 samples)
>          75.0th: 189 (808 samples)
>          90.0th: 216 (501 samples)
>          95.0th: 227 (163 samples)
>          *99.0th: 256 (123 samples)
>          99.5th: 1510 (16 samples)
>          99.9th: 3132 (13 samples)
>          min=3D21, max=3D3286
>
> With 0b0695f2b34a, we get this:
>
> Latency percentiles (usec) runtime 1440 (s) (4480 total samples)
>          50.0th: 147 (2261 samples)
>          75.0th: 182 (1116 samples)
>          90.0th: 205 (671 samples)
>          95.0th: 224 (215 samples)
>          *99.0th: 12240 (173 samples) <=E2=80=94=E2=80=94 much higher p99=
 and up
>          99.5th: 12752 (22 samples)
>          99.9th: 13104 (18 samples)
>          min=3D21, max=3D13172
>
> Since the idea is to fully load the machine with schbench, use schbench
> -t <your_num_cpus/2>, and make sure the box doesn=E2=80=99t have other st=
uff
> running in the background.  I used a VM because it ended up giving more
> consistent results on our kernel test machines, which have some periodic
> noise running in the background.
>
> We=E2=80=99ve tried a few different approaches, but don=E2=80=99t quite h=
ave a solid
> fix yet.  I thought I=E2=80=99d kick off the discussion with my most usef=
ul
> hunks so far:
>
> diff a/kernel/sched/fair.c b/kernel/sched/fair.c
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
>
> -chris