Date: Fri, 4 Mar 2016 07:18:24 -0800
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Ross Green <rgkernel@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        John Stultz <john.stultz@linaro.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Peter Zijlstra <peterz@infradead.org>,
        lkml <linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@kernel.org>,
        Lai Jiangshan <jiangshanlai@gmail.com>, dipankar@in.ibm.com,
        Andrew Morton <akpm@linux-foundation.org>,
        Josh Triplett <josh@joshtriplett.org>, rostedt <rostedt@goodmis.org>,
        David Howells <dhowells@redhat.com>,
        Eric Dumazet <edumazet@google.com>,
        Darren Hart <dvhart@linux.intel.com>,
        =?iso-8859-1?Q?Fr=E9d=E9ric?= Weisbecker <fweisbec@gmail.com>,
        Oleg Nesterov <oleg@redhat.com>, pranith kumar <bobby.prani@gmail.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17
Message-ID: <20160304151824.GR3577@linux.vnet.ibm.com>
Reply-To: paulmck@linux.vnet.ibm.com
References: <20160220063248.GE3522@linux.vnet.ibm.com>
 <CANfgCY2yWp65-f1ujJ_-8yDp8Xp7KPHLJ8u+rfZmz=evkpghVw@mail.gmail.com>
 <CANfgCY0SaXWkCFq=dHGG38AnFd3Rd+wvVGQ6TH9DYow881YUWA@mail.gmail.com>
 <686568926.5862.1456259651418.JavaMail.zimbra@efficios.com>
 <20160223205522.GT3522@linux.vnet.ibm.com>
 <CANfgCY04Cnjoq1jQvQCd48Pt+_4Y9tcG6A7bCHHPWWWL6TFuEQ@mail.gmail.com>
 <CANfgCY3XCmArP9sJsdsQPGygyzWhzAXMS+jnge3_DCNYsONQyg@mail.gmail.com>
 <20160226005638.GV3522@linux.vnet.ibm.com>
 <20160226013557.GA20904@linux.vnet.ibm.com>
 <CANfgCY1GhBrWDYREp6gNzfTYj_P18Vk_p6zmocKfFvw7E2wi1A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CANfgCY1GhBrWDYREp6gNzfTYj_P18Vk_p6zmocKfFvw7E2wi1A@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3371
Lines: 84

On Fri, Mar 04, 2016 at 04:30:12PM +1100, Ross Green wrote:
> On Fri, Feb 26, 2016 at 12:35 PM, Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:

[ . . . ]

> >> OK, so what wakeup path omits the sched_wakeup event?
> >>
> >> The sched_waking event looks to occur once in try_to_wake_up() and
> >> once in try_to_wake_up_local().  Starting with try_to_wake_up():
> >>
> >> o     If the task is ->on_rq, ttwu_remote() is invoked:
> >>
> >>       o       This acquires the runqueue lock, then if
> >>               task_on_rq_queued() invokes ttwu_do_wakeup().  This
> >>               unconditionally does sched_wakeup, so we didn't go that
> >>               way.  (And this path skips the bulk of try_to_wake_up()
> >>               on return.)
> >>
> >>       o       Otherwise, we release the runqueu lock and returns zero.
> >>
> >> o     There is some ordering checking, runqueue selection, and then
> >>       p->state is set to TASK_WAKING.  And we apparently are not getting
> >>       here, either.  But I don't see any other way out.
> >>
> >>       Ignoring this for the moment...
> >>
> >>       We eventually reach to the call to ttwu_queue().
> >>
> >>       o       Here the TTWU_QUEUE path seems to avoid doing a
> >>               sched_wakeup event -- and since we are trying to wake
> >>               CPU 0 from CPU 4, so they don't share cache (x86).
> >>
> >>       o       This invokes ttwu_queue_remote(), which sends an IPI
> >>               unless polling is in effect.  I would need to enable
> >>               trace_sched_wake_idle_without_ipi() to see whether or
> >>               not the IPI was actually sent.
> >>
> >>               If the target CPU was offline, we should have seen the
> >>               cpu_is_offline() WARN_ON().  I suppose that the CPU might
> >>               go offline between the check and the ->send_IPI_mask(),
> >>               but only once.  And we are trying to wakeup on CPU 0
> >>               quite a few times.
> >>
> >>       Any thoughts on what to look for?
> >>
> >> Next, try_to_wake_up_local():
> >>
> >> o     After doing several checks, it does the sched_waking event.
> >>
> >> o     If the task is already queued, it calls ttwu_activate().
> >>
> >> o     It then invokes ttwu_do_wakeup(), which unconditionally
> >>       does the sched_wakeup() event.
> >>
> >>       So this path looks unlikely, even ignoring the fact that
> >>       the waking CPU in the traces above is always different than
> >>       the CPU to be awakened on.
> >>
> >> Any thoughts?
> >>
> >>                                                       Thanx, Paul
> G'day,
> 
> 
> Here is a series of rcu_preempt stall events(5) from linux-4.5-rc6 release.
> 
> Again some testing procedure. boot, run series of brief benchmarks and
> then leave idle.
> The first stall event appeared quite quickly - within hours, the rest
> at what appears to be random intervals after that.
> 
> 
> I thought I might give Daniels patch set a try and see how that goes!

Looks like the same issue from dmesg.

For my part, I added more tracing, which seems to have further decreased
the probability of occurrence.  The sched_wake_idle_without_ipi event
did not appear.

My next step is to try writing a torture test focused specifically on
this issue.  We need a faster reproducer to make decent progress.

							Thanx, Paul