Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755395AbdDLRkv (ORCPT ); Wed, 12 Apr 2017 13:40:51 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:59205 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754620AbdDLRkf (ORCPT ); Wed, 12 Apr 2017 13:40:35 -0400 From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com, "Paul E. McKenney" Subject: [PATCH tip/core/rcu 08/40] rcu: Make call_rcu() do deferred NOCB wakeups Date: Wed, 12 Apr 2017 10:39:53 -0700 X-Mailer: git-send-email 2.5.2 In-Reply-To: <20170412174003.GA23207@linux.vnet.ibm.com> References: <20170412174003.GA23207@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17041217-0024-0000-0000-000002442EE4 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006924; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00846603; UDB=6.00417607; IPR=6.00625041; BA=6.00005286; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015023; XFM=3.00000013; UTC=2017-04-12 17:40:32 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17041217-0025-0000-0000-00004350D088 Message-Id: <1492018825-25634-8-git-send-email-paulmck@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1704120145 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1772 Lines: 42 If a CONFIG_RCU_NOCB_CPUS kernel invokes call_rcu() with interrupts disabled, wakeups must be deferred in order to avoid self-deadlock in the cases where the disabled interrupts are due to scheduler locks being held. In this case, a flag is set and is checked on entry to extended quiescent states (usermode, idle), on exit from the RCU_SOFTIRQ handler, and when the CPU in question goes offline. However, a given CPU could avoid all of those states for a considerable length of time. This commit therefore allows a subsequent invocation of call_rcu() with interrupts enabled to do the wakeup. It also makes the wakeup function clear the deferred-wakeup flag. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 0a62a8f1caac..5a149091d262 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1768,6 +1768,7 @@ static void wake_nocb_leader(struct rcu_data *rdp, bool force) return; if (READ_ONCE(rdp_leader->nocb_leader_sleep) || force) { /* Prior smp_mb__after_atomic() orders against prior enqueue. */ + WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE_NOT); WRITE_ONCE(rdp_leader->nocb_leader_sleep, false); swake_up(&rdp_leader->nocb_wq); } @@ -1853,7 +1854,9 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp, return; } len = atomic_long_read(&rdp->nocb_q_count); - if (old_rhpp == &rdp->nocb_head) { + if (old_rhpp == &rdp->nocb_head || + (rcu_nocb_need_deferred_wakeup(rdp) && + !irqs_disabled_flags(flags))) { if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ wake_nocb_leader(rdp, false); -- 2.5.2