Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755580AbdDLRuI (ORCPT ); Wed, 12 Apr 2017 13:50:08 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:50689 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754312AbdDLRke (ORCPT ); Wed, 12 Apr 2017 13:40:34 -0400 From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com, "Paul E. McKenney" Subject: [PATCH tip/core/rcu 11/40] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure Date: Wed, 12 Apr 2017 10:39:56 -0700 X-Mailer: git-send-email 2.5.2 In-Reply-To: <20170412174003.GA23207@linux.vnet.ibm.com> References: <20170412174003.GA23207@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17041217-0044-0000-0000-0000030316B8 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006924; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00846603; UDB=6.00417608; IPR=6.00625041; BA=6.00005286; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015023; XFM=3.00000013; UTC=2017-04-12 17:40:32 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17041217-0045-0000-0000-00000731195C Message-Id: <1492018825-25634-11-git-send-email-paulmck@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-12_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1704120145 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4477 Lines: 107 The rcu_sched_qs_mask variable is yet another isolated per-CPU variable, so this commit pulls it into the pre-existing rcu_dynticks per-CPU structure. Signed-off-by: Paul E. McKenney --- .../RCU/Design/Data-Structures/Data-Structures.html | 9 ++++++++- kernel/rcu/tree.c | 12 +++++------- kernel/rcu/tree.h | 1 + 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Documentation/RCU/Design/Data-Structures/Data-Structures.html b/Documentation/RCU/Design/Data-Structures/Data-Structures.html index d583c653a703..bf7f266e8888 100644 --- a/Documentation/RCU/Design/Data-Structures/Data-Structures.html +++ b/Documentation/RCU/Design/Data-Structures/Data-Structures.html @@ -1104,6 +1104,7 @@ Its fields are as follows: 1 int dynticks_nesting; 2 int dynticks_nmi_nesting; 3 atomic_t dynticks; + 4 int rcu_sched_qs_mask;

The ->dynticks_nesting field counts the @@ -1117,11 +1118,17 @@ NMIs are counted by the ->dynticks_nmi_nesting field, except that NMIs that interrupt non-dyntick-idle execution are not counted. -

Finally, the ->dynticks field counts the corresponding +

The ->dynticks field counts the corresponding CPU's transitions to and from dyntick-idle mode, so that this counter has an even value when the CPU is in dyntick-idle mode and an odd value otherwise. +

Finally, the ->rcu_sched_qs_mask field is used +to record the fact that the RCU core code would really like to +see a quiescent state from the corresponding CPU. +This flag is checked by RCU's context-switch and cond_resched() +code, which provide a momentary idle sojourn in response. + diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7fa46967021f..315647d4e4cd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -272,8 +272,6 @@ void rcu_bh_qs(void) } } -static DEFINE_PER_CPU(int, rcu_sched_qs_mask); - /* * Steal a bit from the bottom of ->dynticks for idle entry/exit * control. Initially this is for TLB flushing. @@ -464,8 +462,8 @@ static void rcu_momentary_dyntick_idle(void) * Yes, we can lose flag-setting operations. This is OK, because * the flag will be set again after some delay. */ - resched_mask = raw_cpu_read(rcu_sched_qs_mask); - raw_cpu_write(rcu_sched_qs_mask, 0); + resched_mask = raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask); + raw_cpu_write(rcu_dynticks.rcu_sched_qs_mask, 0); /* Find the flavor that needs a quiescent state. */ for_each_rcu_flavor(rsp) { @@ -501,7 +499,7 @@ void rcu_note_context_switch(void) trace_rcu_utilization(TPS("Start context switch")); rcu_sched_qs(); rcu_preempt_note_context_switch(); - if (unlikely(raw_cpu_read(rcu_sched_qs_mask))) + if (unlikely(raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask))) rcu_momentary_dyntick_idle(); for_each_rcu_flavor(rsp) do_nocb_deferred_wakeup(this_cpu_ptr(rsp->rda)); @@ -529,7 +527,7 @@ void rcu_all_qs(void) struct rcu_state *rsp; barrier(); /* Avoid RCU read-side critical sections leaking down. */ - if (unlikely(raw_cpu_read(rcu_sched_qs_mask))) { + if (unlikely(raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask))) { local_irq_save(flags); rcu_momentary_dyntick_idle(); local_irq_restore(flags); @@ -1361,7 +1359,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp, * is set too high, we override with half of the RCU CPU stall * warning delay. */ - rcrmp = &per_cpu(rcu_sched_qs_mask, rdp->cpu); + rcrmp = &per_cpu(rcu_dynticks.rcu_sched_qs_mask, rdp->cpu); if (time_after(jiffies, rdp->rsp->gp_start + jtsq) || time_after(jiffies, rdp->rsp->jiffies_resched)) { if (!(READ_ONCE(*rcrmp) & rdp->rsp->flavor_mask)) { diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 7468b4de7e0c..e298281984dc 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -113,6 +113,7 @@ struct rcu_dynticks { /* Process level is worth LLONG_MAX/2. */ int dynticks_nmi_nesting; /* Track NMI nesting level. */ atomic_t dynticks; /* Even value for idle, else odd. */ + int rcu_sched_qs_mask; /* GP old, need quiescent state. */ #ifdef CONFIG_NO_HZ_FULL_SYSIDLE long long dynticks_idle_nesting; /* irq/process nesting level from idle. */ -- 2.5.2
 
Quick Quiz: