Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 02 Jul 2021 09:33:45 +1000
From:   Nicholas Piggin <npiggin@gmail.com>
Subject: Re: [PATCH] nohz: nohz idle balancing per node
To:     Mel Gorman <mgorman@suse.de>, Peter Zijlstra <peterz@infradead.org>
Cc:     Daniel Bristot de Oliveira <bristot@redhat.com>,
        Ben Segall <bsegall@google.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        Vaidyanathan Srinivasan <svaidy@linux.ibm.com>,
        Vincent Guittot <vincent.guittot@linaro.org>
References: <20210701055323.2199175-1-npiggin@gmail.com>
        <YN2Wav1CSVq+6cS+@hirez.programming.kicks-ass.net>
        <20210701131139.GD3772@suse.de>
In-Reply-To: <20210701131139.GD3772@suse.de>
MIME-Version: 1.0
Message-Id: <1625181736.w1x011tuds.astroid@bobo.none>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Excerpts from Mel Gorman's message of July 1, 2021 11:11 pm:
> On Thu, Jul 01, 2021 at 12:18:18PM +0200, Peter Zijlstra wrote:
>> On Thu, Jul 01, 2021 at 03:53:23PM +1000, Nicholas Piggin wrote:
>> > Currently a single nohz idle CPU is designated to perform balancing on
>> > behalf of all other nohz idle CPUs in the system. Implement a per node
>> > nohz balancer to minimize cross-node memory accesses and runqueue lock
>> > acquisitions.
>> >=20
>> > On a 4 node system, this improves performance by 9.3% on a 'pgbench -N=
'
>> > with 32 clients/jobs (which is about where throughput maxes out due to
>> > IO and contention in postgres).
>>=20
>> Hmm, Suresh tried something like this around 2010 and then we ran into
>> trouble that when once node went completely idle and another node was
>> fully busy, the completely idle node would not run ILB and the node
>> would forever stay idle.
>>=20
>=20
> An effect like that *might* be visible at
> https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-=
nohznuma-v1r1/html/network-tbench/hardy2/
> at the CPU usage heatmaps ordered by topology at the very bottom of
> the page.
>=20
> The heatmap covers all client counts so there are "blocks" of activity fo=
r
> each client count tested. The third block is for 8 thread counts so a nod=
e
> is not fully busy yet.

I'm not sure what I'm looking at. Where are these blocks? Along the x=20
axis?

> However, with the vanilla kernel, there is some
> load on each node but with the patch all the load is on one node. This
> did not happen on the two other test machines so the observation is not
> reliable and could be a total coincidence.

tbench is pretty finicky so it could be.

>=20
> That said, there were some gains but large losses depending on the client
> count across the 3 machines for tbench which is a concern. Other results,
> like pgbench mentioned in the changelog, will not complete until tomorrow
> to see if it is a general pattern or tbench-specific.
>=20
> https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-=
nohznuma-v1r1/html/network-tbench/bing2/
> https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-=
nohznuma-v1r1/html/network-tbench/hardy2/
> https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-=
nohznuma-v1r1/html/network-tbench/marvin2/

All 2-node. How many runs does it do at each clinet count? There's a big=20
regression at one clinet with one of them, but the other two have small=20
gains.

Thanks,
Nick