Date: Thu, 13 Apr 2017 11:19:26 -0700
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, jiangshanlai@gmail.com,
        dipankar@in.ibm.com, akpm@linux-foundation.org,
        mathieu.desnoyers@efficios.com, josh@joshtriplett.org,
        tglx@linutronix.de, rostedt@goodmis.org, dhowells@redhat.com,
        edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com,
        bobby.prani@gmail.com
Subject: Re: [PATCH tip/core/rcu 04/13] rcu: Make RCU_FANOUT_LEAF help text
 more explicit about skew_tick
Reply-To: paulmck@linux.vnet.ibm.com
References: <20170412165441.GA17149@linux.vnet.ibm.com>
 <1492016149-18834-4-git-send-email-paulmck@linux.vnet.ibm.com>
 <20170413091535.r6iw7s3pc2znvl6b@hirez.programming.kicks-ass.net>
 <20170413160332.GZ3956@linux.vnet.ibm.com>
 <20170413161948.ymvzlzhporgmldvn@hirez.programming.kicks-ass.net>
 <20170413165516.GI3956@linux.vnet.ibm.com>
 <20170413170434.xk4zq3p75pu3ubxw@hirez.programming.kicks-ass.net>
 <20170413173100.GL3956@linux.vnet.ibm.com>
 <20170413174631.56ycg545gwbsb4q2@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170413174631.56ycg545gwbsb4q2@hirez.programming.kicks-ass.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20170413181926.GP3956@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1623
Lines: 34

On Thu, Apr 13, 2017 at 07:46:31PM +0200, Peter Zijlstra wrote:
> On Thu, Apr 13, 2017 at 10:31:00AM -0700, Paul E. McKenney wrote:
> > On Thu, Apr 13, 2017 at 07:04:34PM +0200, Peter Zijlstra wrote:
> > > On Thu, Apr 13, 2017 at 09:55:16AM -0700, Paul E. McKenney wrote:
> > > > > > To avoid people tuning huge machines having to wait for me to give
> > > > > > them an answer as to why they are suffering lock contention after
> > > > > > cranking up the value of RCU_FANOUT_LEAF.
> > > 
> > > So is there a good reason to increase FANOUT_LEAF ?
> > 
> > Increasing it reduces the number of rcu_node structures, and thus the
> > number of cache misses during grace-period initialization and cleanup.
> > This has proven necessary in the past on large machines having long
> > memory latencies.  And there are starting to be some pretty big machines
> > running in production, and even for typical commerical workloads.
> 
> Is that perhaps a good moment to look at aligning the cpus in said nodes
> with the cache topology?

We have been here before, haven't we?  Over and over again.  ;-)

As always...

First get me some system-level data showing that the current layout is
causing a real problem.  RCU's fastpath code doesn't come anywhere near
the rcu_node tree, so in the absence of such data, I of course remain
quite doubtful that there is a real need.  And painfully aware of the
required increase in complexity.

But if there is a real need demonstrated by real system-level data,
I will of course make the needed changes, as I have done many times in
the past in response to other requests.

							Thanx, Paul