Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 23 Oct 2018 00:05:09 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     Steven Sistare <steven.sistare@oracle.com>
Cc:     mingo@redhat.com, subhra.mazumdar@oracle.com,
        dhaval.giani@oracle.com, rohit.k.jain@oracle.com,
        daniel.m.jordan@oracle.com, pavel.tatashin@microsoft.com,
        matt@codeblueprint.co.uk, umgwanakikbuti@gmail.com,
        riel@redhat.com, jbacik@fb.com, juri.lelli@redhat.com,
        linux-kernel@vger.kernel.org, srikar@linux.vnet.ibm.com
Subject: Re: [PATCH 09/10] sched/fair: disable stealing if too many NUMA nodes
Message-ID: <20181022220509.GC3109@worktop.c.hoisthospitality.com>
References: <1540220381-424433-1-git-send-email-steven.sistare@oracle.com>
 <1540220381-424433-10-git-send-email-steven.sistare@oracle.com>
 <20181022170604.GG3117@worktop.programming.kicks-ass.net>
 <1356a3cc-a9f2-6593-4737-1b20fc5daa75@oracle.com>
 <274b79fd-b001-3d30-3773-7f01940ca8a6@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <274b79fd-b001-3d30-3773-7f01940ca8a6@oracle.com>
User-Agent: Mutt/1.5.22.1 (2013-10-16)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Oct 22, 2018 at 03:21:20PM -0400, Steven Sistare wrote:
> On 10/22/2018 2:47 PM, Steven Sistare wrote:
> > On 10/22/2018 1:06 PM, Peter Zijlstra wrote:
> >> On Mon, Oct 22, 2018 at 07:59:40AM -0700, Steve Sistare wrote:
> >>> The STEAL feature causes regressions on hackbench on larger NUMA systems,
> >>> so disable it on systems with more than sched_steal_node_limit nodes
> >>> (default 2).
> >>
> >> How come? From a quick read the stealing is per LLC, where do we steal
> >> across nodes?
> > 
> > See the complete explanation in this patch.  It is deeper than can be gleaned
> > from a quick read.
> 
> I should have said a bit more.  Your quick take on stealing is correct, we do
> not steal across nodes.  However, stealing reduces average run queue length which 
> influences wake_affine migrations.  Now see the complete explanation.

Right; read a bit more just now.

hackbench is a fairly poor benchmark for numa performance. One that
comes to mind is multi wharehouse specjbb stuff (assuming you have numa
balance enabled of course).