Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 15 May 2020 14:51:28 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     Mel Gorman <mgorman@techsingularity.net>
Cc:     Jirka Hladky <jhladky@redhat.com>, Phil Auld <pauld@redhat.com>,
        Ingo Molnar <mingo@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Hillf Danton <hdanton@sina.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Douglas Shakshober <dshaks@redhat.com>,
        Waiman Long <longman@redhat.com>,
        Joe Mario <jmario@redhat.com>, Bill Gray <bgray@redhat.com>
Subject: Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load
 balancer v6
Message-ID: <20200515125128.GU2957@hirez.programming.kicks-ass.net>
References: <20200507155422.GD3758@techsingularity.net>
 <CAE4VaGCDTeE16nNmSS8fGzCBvHsO=qkJAW6yDiORAxgsPi-Ziw@mail.gmail.com>
 <20200508092212.GE3758@techsingularity.net>
 <CAE4VaGC_v6On-YvqdTwAWu3Mq4ofiV0pLov-QpV+QHr_SJr+Rw@mail.gmail.com>
 <CAE4VaGDQWPePtmtCZP=ROYW1KPxtPhGDrxqy2QbirHGJdwk4=w@mail.gmail.com>
 <20200513153023.GF3758@techsingularity.net>
 <20200514153122.GE2978@hirez.programming.kicks-ass.net>
 <20200515084740.GJ3758@techsingularity.net>
 <20200515112815.GT2957@hirez.programming.kicks-ass.net>
 <20200515122231.GL3758@techsingularity.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20200515122231.GL3758@techsingularity.net>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, May 15, 2020 at 01:22:31PM +0100, Mel Gorman wrote:
> On Fri, May 15, 2020 at 01:28:15PM +0200, Peter Zijlstra wrote:

> > > +			if (val & _TIF_POLLING_NRFLAG)
> > > +				goto activate;
> > 
> > I'm completely confused... the result here is that if you're polling you
> > do _NOT_ queue on the wake_list, but instead immediately enqueue.
> > 
> > (which kinda makes sense, since if the remote CPU is idle, it doesn't
> > have these lines in its cache anyway)
> > 
> 
> Crap, I rushed this and severely confused myself about what is going

Hehe, and here I though I was confused :-)

> on. It is definitely the case that flipping this check does not give
> any benefit. The patch shows a benefit but I'm failing to understand
> exactly why. How I ended up here was perf indicating a lot of time spent
> on smp_cond_load_acquire() which made me look closely at ttwu_remote()
> and looking at function graphs to compare the different types of wakeups
> and their timings.

So the raisin we did this remote wakeup thing in the first place was
that Oracle was having very heavy rq->lock cache-line contention. By
farming off the enqueue to the CPU that was going to run the task
anyway, the rq->lock (and the other runqueue structure lines) could stay
in the CPU that was using them (hard). Less cacheline ping-pong, more
win.

The observation here is that if a CPU is idle, it's rq will not be
contended.