Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 28 Jun 2019 11:22:16 -0700
From:   "Paul E. McKenney" <paulmck@linux.ibm.com>
To:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc:     Joel Fernandes <joel@joelfernandes.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        rcu <rcu@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Lai Jiangshan <jiangshanlai@gmail.com>
Subject: Re: [RFC] Deadlock via recursive wakeup via RCU with threadirqs
Reply-To: paulmck@linux.ibm.com
References: <20190627153031.GA249127@google.com>
 <20190627155506.GU26519@linux.ibm.com>
 <CAEXW_YSEN_OL3ftTLN=M-W70WSuCgHJqU-R9VhS=A3uVj_AL+A@mail.gmail.com>
 <20190627173831.GW26519@linux.ibm.com>
 <20190627181638.GA209455@google.com>
 <20190627184107.GA26519@linux.ibm.com>
 <20190628164008.GB240964@google.com>
 <20190628164559.GC240964@google.com>
 <20190628173011.GX26519@linux.ibm.com>
 <20190628174545.pwgwi3wxl2eapkvm@linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20190628174545.pwgwi3wxl2eapkvm@linutronix.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20190628182216.GY26519@linux.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, Jun 28, 2019 at 07:45:45PM +0200, Sebastian Andrzej Siewior wrote:
> On 2019-06-28 10:30:11 [-0700], Paul E. McKenney wrote:
> > > I believe the .blocked field remains set even though we are not any more in a
> > > reader section because of deferred processing of the blocked lists that you
> > > mentioned yesterday.
> > 
> > That can indeed happen.  However, in current -rcu, that would mean
> > that .deferred_qs is also set, which (if in_irq()) would prevent
> > the raise_softirq_irqsoff() from being invoked.  Which was why I was
> > asking the questions about whether in_irq() returns true within threaded
> > interrupts yesterday.  If it does, I need to find if there is some way
> > of determining whether rcu_read_unlock_special() is being called from
> > a threaded interrupt in order to suppress the call to raise_softirq()
> > in that case.
> 
> Please not that:
> | void irq_exit(void)
> | {
> |…
> in_irq() returns true
> |         preempt_count_sub(HARDIRQ_OFFSET);
> in_irq() returns false
> |         if (!in_interrupt() && local_softirq_pending())
> |                 invoke_softirq();
> 
> -> invoke_softirq() does
> |        if (!force_irqthreads) {
> |                 __do_softirq();
> |         } else {
> |                 wakeup_softirqd();
> |         }
> 
> so for `force_irqthreads' rcu_read_unlock_special() within
> wakeup_softirqd() will see false.

OK, fair point.  How about the following instead, again on -rcu?

Here is the rationale for the new version of the "if" statement:

1.	irqs_were_disabled:  If interrupts are enabled, we should
	instead let the upcoming irq_enable()/local_bh_enable()
	do the rescheduling for us.
2.	use_softirq: If we aren't using softirq, then
	raise_softirq_irqoff() will be unhelpful.
3a.	in_interrupt(): If this returns true, the subsequent
	call to raise_softirq_irqoff() is guaranteed not to
	do a wakeup, so that call will be both very cheap and
	quite safe.
3b.	Otherwise, if !in_interrupt(), if exp (an expedited RCU grace
	period is being blocked), then incurring wakeup overhead
	is worthwhile, and if also !.deferred_qs then scheduler locks
	cannot be held so the wakeup will be safe.

Does that make more sense?

							Thanx, Paul

------------------------------------------------------------------------

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 82c925df1d92..83333cfe8707 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -624,8 +624,9 @@ static void rcu_read_unlock_special(struct task_struct *t)
 		      (rdp->grpmask & rnp->expmask) ||
 		      tick_nohz_full_cpu(rdp->cpu);
 		// Need to defer quiescent state until everything is enabled.
-		if ((exp || in_irq()) && irqs_were_disabled && use_softirq &&
-		    (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
+		if (irqs_were_disabled && use_softirq &&
+		    (in_interrupt() ||
+		     (exp && !t->rcu_read_unlock_special.b.deferred_qs))) {
 			// Using softirq, safe to awaken, and we get
 			// no help from enabling irqs, unlike bh/preempt.
 			raise_softirq_irqoff(RCU_SOFTIRQ);