Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 19 Sep 2018 15:11:40 -0700
From:   "Paul E. McKenney" <paulmck@linux.ibm.com>
To:     Tejun Heo <tj@kernel.org>
Cc:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        linux-kernel@vger.kernel.org, Boqun Feng <boqun.feng@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>,
        "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>,
        tglx@linutronix.de, Steven Rostedt <rostedt@goodmis.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Lai Jiangshan <jiangshanlai@gmail.com>
Subject: Re: [PATCH] rcu: Use cpus_read_lock() while looking at
 cpu_online_mask
Reply-To: paulmck@linux.ibm.com
References: <20180910135615.tr3cvipwbhq6xug4@linutronix.de>
 <20180911160532.GJ4225@linux.vnet.ibm.com>
 <20180911162142.cc3vgook2gctus4c@linutronix.de>
 <20180911170222.GO4225@linux.vnet.ibm.com>
 <20180919205521.GE902964@devbig004.ftw2.facebook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180919205521.GE902964@devbig004.ftw2.facebook.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20180919221140.GH4222@linux.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Sep 19, 2018 at 01:55:21PM -0700, Tejun Heo wrote:
> Hello,
> 
> On Tue, Sep 11, 2018 at 10:02:22AM -0700, Paul E. McKenney wrote:
> > > Doesn't work for me because it is still within the preempt-disable
> > > section :/.
> > > Would it work to use WORK_CPU_UNBOUND? As far as I understand it, the
> > > CPU number does not matter, you just want to spread it across multiple
> > > CPUs in the NUMA case.
> > 
> > Locality is a good thing, but yes, something like this?
> > 
> > 	if (!IS_ENABLED(CONFIG_PREEMPT_RT) && /* or whatever it is called */
> > 	    unlikely(cpu > rnp->grphi - rnp->grplo))
> > 
> > Another approach that might be better longer term would be to have a
> > workqueue interface that treats the specified CPU as a suggestion,
> > and silently switches to WORK_CPU_UNBOUND if there is any problem
> > whatsoever with the specified CPU.  Tejun, Lai, thoughts?
> 
> Unbound workqueue is NUMA-affine by default, so using it by default
> might not harm anything.

OK, so the above workaround would function correctly on -rt, thank you!

Sebastian, is there a counterpart to CONFIG_PREEMPT_RT already in
mainline?  If so, I would be happy to make mainline safe for -rt.

>                           Also, per-cpu work items get unbound from
> the cpu if the cpu goes down while the work item is running or queued,
> so it might just work already.

There are race conditions where the work item is queued at an inopportune
time during the offline process, resulting in a splat, hence the need
for a check with preemption disabled in order to synchronize with the
synchronize_sched() in the offline process.

							Thanx, Paul