Date: Sun, 27 Mar 2016 22:45:59 +0200
From: Peter Zijlstra <peterz@infradead.org>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        "Chatre, Reinette" <reinette.chatre@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        Josh Triplett <josh@joshtriplett.org>, Ross Green <rgkernel@gmail.com>,
        John Stultz <john.stultz@linaro.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        lkml <linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@kernel.org>,
        Lai Jiangshan <jiangshanlai@gmail.com>, dipankar@in.ibm.com,
        Andrew Morton <akpm@linux-foundation.org>,
        rostedt <rostedt@goodmis.org>, David Howells <dhowells@redhat.com>,
        Eric Dumazet <edumazet@google.com>,
        Darren Hart <dvhart@linux.intel.com>,
        =?iso-8859-1?Q?Fr=E9d=E9ric?= Weisbecker <fweisbec@gmail.com>,
        Oleg Nesterov <oleg@redhat.com>, pranith kumar <bobby.prani@gmail.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17
Message-ID: <20160327204559.GV6356@twins.programming.kicks-ass.net>
References: <20160318235641.GH4287@linux.vnet.ibm.com>
 <0D818C7A2259ED42912C1E04120FDE26712E676E@ORSMSX111.amr.corp.intel.com>
 <20160325214623.GR4287@linux.vnet.ibm.com>
 <1370753660.36931.1458995371427.JavaMail.zimbra@efficios.com>
 <20160326152816.GW4287@linux.vnet.ibm.com>
 <20160326184940.GA23851@linux.vnet.ibm.com>
 <706246733.37102.1459030977316.JavaMail.zimbra@efficios.com>
 <20160327013456.GX4287@linux.vnet.ibm.com>
 <702204510.37291.1459086535844.JavaMail.zimbra@efficios.com>
 <20160327154018.GA4287@linux.vnet.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160327154018.GA4287@linux.vnet.ibm.com>
User-Agent: Mutt/1.5.21 (2012-12-30)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1709
Lines: 39

On Sun, Mar 27, 2016 at 08:40:18AM -0700, Paul E. McKenney wrote:
> Oh, and the patch I am running with is below.  I am running x86, and so
> some other architectures would of course need the corresponding patch
> on that architecture.

> -#define TIF_POLLING_NRFLAG	21	/* idle is polling for TIF_NEED_RESCHED */
> +/* #define TIF_POLLING_NRFLAG	21	 idle is polling for TIF_NEED_RESCHED */

x86 is the only arch that really uses this heavily IIRC.

Most of the other archs need interrupts to wake up remote cores.

So what we try to do is avoid sending IPIs when the CPU is idle, for the
remote wakeup case we use set_nr_if_polling() which sets
TIF_NEED_RESCHED if TIF_POLLING_NRFLAG was set. If it wasn't, we'll send
the IPI. Otherwise we rely on the idle loop to do sched_ttwu_pending()
when it breaks out of loop due to TIF_NEED_RESCHED.

But, you need hotplug for this to happen, right?

We should not be migrating towards, or waking on, CPUs no longer present
in cpu_active_map, and there is a rcu/sched_sync() after clearing that
bit. Furthermore, migration_call() does a sched_ttwu_pending() (waking
any remaining stragglers) before we migrate all runnable tasks off the
dying CPU.


The other interesting case would be resched_cpu(), which uses
set_nr_and_not_polling() to kick a remote cpu to call schedule(). It
atomically sets TIF_NEED_RESCHED and returns if TIF_POLLING_NRFLAG was
not set. If indeed not, it will send an IPI.

This assumes the idle 'exit' path will do the same as the IPI does; and
if you look at cpu_idle_loop() it does indeed do both
preempt_fold_need_resched() and sched_ttwu_pending().

Note that one cannot rely on irq_enter()/irq_exit() being called for the
scheduler IPI.