Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 20 Mar 2020 15:22:51 +0000
From:   Mel Gorman <mgorman@techsingularity.net>
To:     Jirka Hladky <jhladky@redhat.com>
Cc:     Phil Auld <pauld@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Hillf Danton <hdanton@sina.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load
 balancer v6
Message-ID: <20200320152251.GC3818@techsingularity.net>
References: <20200224095223.13361-1-mgorman@techsingularity.net>
 <20200309191233.GG10065@pauld.bos.csb>
 <20200309203625.GU3818@techsingularity.net>
 <20200312095432.GW3818@techsingularity.net>
 <CAE4VaGA4q4_qfC5qe3zaLRfiJhvMaSb2WADgOcQeTwmPvNat+A@mail.gmail.com>
 <20200312155640.GX3818@techsingularity.net>
 <CAE4VaGD8DUEi6JnKd8vrqUL_8HZXnNyHMoK2D+1-F5wo+5Z53Q@mail.gmail.com>
 <20200312214736.GA3818@techsingularity.net>
 <CAE4VaGCfDpu0EuvHNHwDGbR-HNBSAHY=yu3DJ33drKgymMTTOw@mail.gmail.com>
 <CAE4VaGC09OfU2zXeq2yp_N0zXMbTku5ETz0KEocGi-RSiKXv-w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-15
Content-Disposition: inline
In-Reply-To: <CAE4VaGC09OfU2zXeq2yp_N0zXMbTku5ETz0KEocGi-RSiKXv-w@mail.gmail.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, Mar 20, 2020 at 03:37:44PM +0100, Jirka Hladky wrote:
> Hi Mel,
> 
> just a quick update. I have increased the testing coverage and other tests
> from the NAS shows a big performance drop for the low number of threads as
> well:
> 
> sp_C_x - show still the biggest drop upto 50%
> bt_C_x - performance drop upto 40%
> ua_C_x - performance drop upto 30%
> 

MPI or OMP and what is a low thread count? For MPI at least, I saw a 0.4%
gain on an 4-node machine for bt_C and a 3.88% regression on 8-nodes. I
think it must be OMP you are using because I found I had to disable UA
for MPI at some point in the past for reasons I no longer remember.

> My point is that the performance drop for the low number of threads is more
> common than we have initially thought.
> 
> Let me know what you need more data.
> 

I just a clarification on the thread count and a confirmation it's OMP. For
MPI, I did note that some of the other NAS kernels shows a slight dip but
it was nowhere near as severe as SP and the problem was the same as more --
two or more tasks stayed on the same node without spreading out because
there was no pressure to do so. There was enough CPU and memory capacity
with no obvious pattern that could be used to spread the load wide early.

One possibility would be to spread wide always at clone time and assume
wake_affine will pull related tasks but it's fragile because it breaks
if the cloned task execs and then allocates memory from a remote node
only to migrate to a local node immediately.

-- 
Mel Gorman
SUSE Labs