Hello!
This series takes advantage of callback numbering to simplify RCU's
grace-period machinery, in some cases also reducing the number of
lock acquisitions (though the resulting change in performance is not
perceptible). The individual patches are as follows:
1. Move code to make way for the code-combining in later patches.
This commit makes no changes, just moves code.
2. Make __note_new_gpnum() also check for the ends of prior grace
periods, thus eliminating the earlier possibility of a given
CPU becoming aware of the start of the next grace period before
becoming aware of the end of the previous grace period. Yes,
the code did handle this correctly, but now it doesn't need to.
More important, now I don't need to think about how it handles
this correctly.
3. Rename note_new_gpnum() to note_gp_changes() in preparation for
later merge of rcu_process_gp_end() into this function.
4. Change calls to rcu_process_gp_end() to instead call
note_gp_changes(), and also remove the now-used rcu_process_gp_end().
5. Remove duplicate code by merging __rcu_process_gp_end() into
__note_gp_changes().
6. Eliminate now-redundant call to check_for_new_grace_period(). This
leaves only a single caller, so inline check_for_new_grace_period().
7. Given that rcu_start_gp_per_cpu() is a trivial wrapper function
with only one caller, inline it into its sole remaining call site.
8. Eliminate now-redundant call to note_gp_changes().
Thanx, Paul
b/kernel/rcutree.c | 262 ++++++++++++++--------------------------------
b/kernel/rcutree_plugin.h | 2
2 files changed, 85 insertions(+), 179 deletions(-)
From: "Paul E. McKenney" <[email protected]>
The current implementation can detect the beginning of a new grace period
before noting the end of a previous grace period. Although the current
implementation correctly handles this sort of nonsense, it would be
good to reduce RCU's state space by making such nonsense unnecessary,
which is now possible thanks to the fact that RCU's callback groups are
now numbered.
This commit therefore makes __note_new_gpnum() invoke
__rcu_process_gp_end() in order to note the ends of prior grace
periods before noting the beginnings of new grace periods.
Of course, this now means that note_new_gpnum() notes both the
beginnings and ends of grace periods, and could therefore be
used in place of rcu_process_gp_end(). But that is a job for
later commits.
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 8ebc3ff..a57bac3 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1299,6 +1299,9 @@ __rcu_process_gp_end(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
*/
static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
{
+ /* Handle the ends of any preceding grace periods first. */
+ __rcu_process_gp_end(rsp, rnp, rdp);
+
if (rdp->gpnum != rnp->gpnum) {
/*
* If the current grace period is waiting for this CPU,
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
One of the calls to check_for_new_grace_period() is now redundant due to
an immediately preceding call to note_gp_changes(). Eliminating this
redundant call leaves a single caller, which is simpler if inlined.
This commit therefore eliminates the redundant call and inlines the
body of check_for_new_grace_period() into the single remaining call site.
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 25 +++----------------------
1 file changed, 3 insertions(+), 22 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index ca07f2d..f6cf5e1 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1298,26 +1298,6 @@ static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)
}
/*
- * Did someone else start a new RCU grace period start since we last
- * checked? Update local state appropriately if so. Must be called
- * on the CPU corresponding to rdp.
- */
-static int
-check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
-{
- unsigned long flags;
- int ret = 0;
-
- local_irq_save(flags);
- if (rdp->gpnum != rsp->gpnum) {
- note_gp_changes(rsp, rdp);
- ret = 1;
- }
- local_irq_restore(flags);
- return ret;
-}
-
-/*
* Do per-CPU grace-period initialization for running CPU. The caller
* must hold the lock of the leaf rcu_node structure corresponding to
* this CPU.
@@ -1727,8 +1707,10 @@ static void
rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
{
/* If there is now a new grace period, record and return. */
- if (check_for_new_grace_period(rsp, rdp))
+ if (rdp->gpnum != rsp->gpnum) {
+ note_gp_changes(rsp, rdp);
return;
+ }
/*
* Does this CPU still need to do its part for current grace period?
@@ -2280,7 +2262,6 @@ static void __call_rcu_core(struct rcu_state *rsp, struct rcu_data *rdp,
/* Are we ignoring a completed grace period? */
note_gp_changes(rsp, rdp);
- check_for_new_grace_period(rsp, rdp);
/* Start a new grace period if one not already started. */
if (!rcu_gp_in_progress(rsp)) {
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
This commit eliminates some duplicated code by merging
__rcu_process_gp_end() into __note_gp_changes().
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 48 ++++++------------------------------------------
1 file changed, 6 insertions(+), 42 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 9040e0f..ca07f2d 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1244,18 +1244,16 @@ static void rcu_advance_cbs(struct rcu_state *rsp, struct rcu_node *rnp,
}
/*
- * Advance this CPU's callbacks, but only if the current grace period
- * has ended. This may be called only from the CPU to whom the rdp
- * belongs. In addition, the corresponding leaf rcu_node structure's
- * ->lock must be held by the caller, with irqs disabled.
+ * Update CPU-local rcu_data state to record the beginnings and ends of
+ * grace periods. The caller must hold the ->lock of the leaf rcu_node
+ * structure corresponding to the current CPU, and must have irqs disabled.
*/
-static void
-__rcu_process_gp_end(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
+static void __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
{
- /* Did another grace period end? */
+ /* Handle the ends of any preceding grace periods first. */
if (rdp->completed == rnp->completed) {
- /* No, so just accelerate recent callbacks. */
+ /* No grace period end, so just accelerate recent callbacks. */
rcu_accelerate_cbs(rsp, rnp, rdp);
} else {
@@ -1266,41 +1264,7 @@ __rcu_process_gp_end(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
/* Remember that we saw this grace-period completion. */
rdp->completed = rnp->completed;
trace_rcu_grace_period(rsp->name, rdp->gpnum, "cpuend");
-
- /*
- * If we were in an extended quiescent state, we may have
- * missed some grace periods that others CPUs handled on
- * our behalf. Catch up with this state to avoid noting
- * spurious new grace periods. If another grace period
- * has started, then rnp->gpnum will have advanced, so
- * we will detect this later on. Of course, any quiescent
- * states we found for the old GP are now invalid.
- */
- if (ULONG_CMP_LT(rdp->gpnum, rdp->completed)) {
- rdp->gpnum = rdp->completed;
- rdp->passed_quiesce = 0;
- }
-
- /*
- * If RCU does not need a quiescent state from this CPU,
- * then make sure that this CPU doesn't go looking for one.
- */
- if ((rnp->qsmask & rdp->grpmask) == 0)
- rdp->qs_pending = 0;
}
-}
-
-/*
- * Update CPU-local rcu_data state to record the newly noticed grace period.
- * This is used both when we started the grace period and when we notice
- * that someone else started the grace period. The caller must hold the
- * ->lock of the leaf rcu_node structure corresponding to the current CPU,
- * and must have irqs disabled.
- */
-static void __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
-{
- /* Handle the ends of any preceding grace periods first. */
- __rcu_process_gp_end(rsp, rnp, rdp);
if (rdp->gpnum != rnp->gpnum) {
/*
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
The __rcu_process_callbacks() invokes note_gp_changes() immediately
before invoking rcu_check_quiescent_state(), which conditionally
invokes that same function. This commit therefore eliminates the
call to note_gp_changes() in __rcu_process_callbacks() in favor of
making unconditional to call from rcu_check_quiescent_state() to
note_gp_changes().
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 0848341..12094bd 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1694,11 +1694,8 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp)
static void
rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
{
- /* If there is now a new grace period, record and return. */
- if (rdp->gpnum != rsp->gpnum) {
- note_gp_changes(rsp, rdp);
- return;
- }
+ /* Check for grace-period ends and beginnings. */
+ note_gp_changes(rsp, rdp);
/*
* Does this CPU still need to do its part for current grace period?
@@ -2162,9 +2159,6 @@ __rcu_process_callbacks(struct rcu_state *rsp)
WARN_ON_ONCE(rdp->beenonline == 0);
- /* Handle the end of a grace period that some other CPU ended. */
- note_gp_changes(rsp, rdp);
-
/* Update RCU state based on any recent quiescent states. */
rcu_check_quiescent_state(rsp, rdp);
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
Given the changes that introduce note_gp_change(), rcu_start_gp_per_cpu()
is now a trivial wrapper function with only one caller. This commit
therefore inlines it into its sole call site.
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 14 +-------------
1 file changed, 1 insertion(+), 13 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index f6cf5e1..0848341 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1298,18 +1298,6 @@ static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)
}
/*
- * Do per-CPU grace-period initialization for running CPU. The caller
- * must hold the lock of the leaf rcu_node structure corresponding to
- * this CPU.
- */
-static void
-rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
-{
- /* Set state so that this CPU will detect the next quiescent state. */
- __note_gp_changes(rsp, rnp, rdp);
-}
-
-/*
* Initialize a new grace period.
*/
static int rcu_gp_init(struct rcu_state *rsp)
@@ -1357,7 +1345,7 @@ static int rcu_gp_init(struct rcu_state *rsp)
WARN_ON_ONCE(rnp->completed != rsp->completed);
ACCESS_ONCE(rnp->completed) = rsp->completed;
if (rnp == rdp->mynode)
- rcu_start_gp_per_cpu(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
rcu_preempt_boost_start_gp(rnp);
trace_rcu_grace_period_init(rsp->name, rnp->gpnum,
rnp->level, rnp->grplo,
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
Because note_new_gpnum() now also checks for the ends of old grace periods,
this commit changes its name to note_gp_changes(). Later commits will merge
rcu_process_gp_end() into note_gp_changes().
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index a57bac3..dded193 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1297,7 +1297,7 @@ __rcu_process_gp_end(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
* ->lock of the leaf rcu_node structure corresponding to the current CPU,
* and must have irqs disabled.
*/
-static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
+static void __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
{
/* Handle the ends of any preceding grace periods first. */
__rcu_process_gp_end(rsp, rnp, rdp);
@@ -1316,19 +1316,20 @@ static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct
}
}
-static void note_new_gpnum(struct rcu_state *rsp, struct rcu_data *rdp)
+static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)
{
unsigned long flags;
struct rcu_node *rnp;
local_irq_save(flags);
rnp = rdp->mynode;
- if (rdp->gpnum == ACCESS_ONCE(rnp->gpnum) || /* outside lock. */
+ if ((rdp->gpnum == ACCESS_ONCE(rnp->gpnum) &&
+ rdp->completed == ACCESS_ONCE(rnp->completed)) || /* w/out lock. */
!raw_spin_trylock(&rnp->lock)) { /* irqs already off, so later. */
local_irq_restore(flags);
return;
}
- __note_new_gpnum(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
raw_spin_unlock_irqrestore(&rnp->lock, flags);
}
@@ -1367,7 +1368,7 @@ check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
local_irq_save(flags);
if (rdp->gpnum != rsp->gpnum) {
- note_new_gpnum(rsp, rdp);
+ note_gp_changes(rsp, rdp);
ret = 1;
}
local_irq_restore(flags);
@@ -1386,7 +1387,7 @@ rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
__rcu_process_gp_end(rsp, rnp, rdp);
/* Set state so that this CPU will detect the next quiescent state. */
- __note_new_gpnum(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
}
/*
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
The addition of callback numbering allows combining the detection of the
ends of old grace periods and the beginnings of new grace periods. This
commit moves code to set the stage for this combining.
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 118 +++++++++++++++++++++++++++----------------------------
1 file changed, 59 insertions(+), 59 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index bc3eac5..8ebc3ff 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -975,65 +975,6 @@ void rcu_cpu_stall_reset(void)
}
/*
- * Update CPU-local rcu_data state to record the newly noticed grace period.
- * This is used both when we started the grace period and when we notice
- * that someone else started the grace period. The caller must hold the
- * ->lock of the leaf rcu_node structure corresponding to the current CPU,
- * and must have irqs disabled.
- */
-static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
-{
- if (rdp->gpnum != rnp->gpnum) {
- /*
- * If the current grace period is waiting for this CPU,
- * set up to detect a quiescent state, otherwise don't
- * go looking for one.
- */
- rdp->gpnum = rnp->gpnum;
- trace_rcu_grace_period(rsp->name, rdp->gpnum, "cpustart");
- rdp->passed_quiesce = 0;
- rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
- zero_cpu_stall_ticks(rdp);
- }
-}
-
-static void note_new_gpnum(struct rcu_state *rsp, struct rcu_data *rdp)
-{
- unsigned long flags;
- struct rcu_node *rnp;
-
- local_irq_save(flags);
- rnp = rdp->mynode;
- if (rdp->gpnum == ACCESS_ONCE(rnp->gpnum) || /* outside lock. */
- !raw_spin_trylock(&rnp->lock)) { /* irqs already off, so later. */
- local_irq_restore(flags);
- return;
- }
- __note_new_gpnum(rsp, rnp, rdp);
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
-}
-
-/*
- * Did someone else start a new RCU grace period start since we last
- * checked? Update local state appropriately if so. Must be called
- * on the CPU corresponding to rdp.
- */
-static int
-check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
-{
- unsigned long flags;
- int ret = 0;
-
- local_irq_save(flags);
- if (rdp->gpnum != rsp->gpnum) {
- note_new_gpnum(rsp, rdp);
- ret = 1;
- }
- local_irq_restore(flags);
- return ret;
-}
-
-/*
* Initialize the specified rcu_data structure's callback list to empty.
*/
static void init_callback_list(struct rcu_data *rdp)
@@ -1350,6 +1291,45 @@ __rcu_process_gp_end(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
}
/*
+ * Update CPU-local rcu_data state to record the newly noticed grace period.
+ * This is used both when we started the grace period and when we notice
+ * that someone else started the grace period. The caller must hold the
+ * ->lock of the leaf rcu_node structure corresponding to the current CPU,
+ * and must have irqs disabled.
+ */
+static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
+{
+ if (rdp->gpnum != rnp->gpnum) {
+ /*
+ * If the current grace period is waiting for this CPU,
+ * set up to detect a quiescent state, otherwise don't
+ * go looking for one.
+ */
+ rdp->gpnum = rnp->gpnum;
+ trace_rcu_grace_period(rsp->name, rdp->gpnum, "cpustart");
+ rdp->passed_quiesce = 0;
+ rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
+ zero_cpu_stall_ticks(rdp);
+ }
+}
+
+static void note_new_gpnum(struct rcu_state *rsp, struct rcu_data *rdp)
+{
+ unsigned long flags;
+ struct rcu_node *rnp;
+
+ local_irq_save(flags);
+ rnp = rdp->mynode;
+ if (rdp->gpnum == ACCESS_ONCE(rnp->gpnum) || /* outside lock. */
+ !raw_spin_trylock(&rnp->lock)) { /* irqs already off, so later. */
+ local_irq_restore(flags);
+ return;
+ }
+ __note_new_gpnum(rsp, rnp, rdp);
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+}
+
+/*
* Advance this CPU's callbacks, but only if the current grace period
* has ended. This may be called only from the CPU to whom the rdp
* belongs.
@@ -1372,6 +1352,26 @@ rcu_process_gp_end(struct rcu_state *rsp, struct rcu_data *rdp)
}
/*
+ * Did someone else start a new RCU grace period start since we last
+ * checked? Update local state appropriately if so. Must be called
+ * on the CPU corresponding to rdp.
+ */
+static int
+check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
+{
+ unsigned long flags;
+ int ret = 0;
+
+ local_irq_save(flags);
+ if (rdp->gpnum != rsp->gpnum) {
+ note_new_gpnum(rsp, rdp);
+ ret = 1;
+ }
+ local_irq_restore(flags);
+ return ret;
+}
+
+/*
* Do per-CPU grace-period initialization for running CPU. The caller
* must hold the lock of the leaf rcu_node structure corresponding to
* this CPU.
--
1.8.1.5
From: "Paul E. McKenney" <[email protected]>
Because note_gp_changes() now incorporates rcu_process_gp_end() function,
this commit switches to the former and eliminates the latter. In
addition, this commit changes external calls from __rcu_process_gp_end()
to __note_gp_changes().
Signed-off-by: Paul E. McKenney <[email protected]>
---
kernel/rcutree.c | 31 +++----------------------------
kernel/rcutree_plugin.h | 2 +-
2 files changed, 4 insertions(+), 29 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index dded193..9040e0f 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1334,28 +1334,6 @@ static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)
}
/*
- * Advance this CPU's callbacks, but only if the current grace period
- * has ended. This may be called only from the CPU to whom the rdp
- * belongs.
- */
-static void
-rcu_process_gp_end(struct rcu_state *rsp, struct rcu_data *rdp)
-{
- unsigned long flags;
- struct rcu_node *rnp;
-
- local_irq_save(flags);
- rnp = rdp->mynode;
- if (rdp->completed == ACCESS_ONCE(rnp->completed) || /* outside lock. */
- !raw_spin_trylock(&rnp->lock)) { /* irqs already off, so later. */
- local_irq_restore(flags);
- return;
- }
- __rcu_process_gp_end(rsp, rnp, rdp);
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
-}
-
-/*
* Did someone else start a new RCU grace period start since we last
* checked? Update local state appropriately if so. Must be called
* on the CPU corresponding to rdp.
@@ -1383,9 +1361,6 @@ check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp)
static void
rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp)
{
- /* Prior grace period ended, so advance callbacks for current CPU. */
- __rcu_process_gp_end(rsp, rnp, rdp);
-
/* Set state so that this CPU will detect the next quiescent state. */
__note_gp_changes(rsp, rnp, rdp);
}
@@ -1521,7 +1496,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)
ACCESS_ONCE(rnp->completed) = rsp->gpnum;
rdp = this_cpu_ptr(rsp->rda);
if (rnp == rdp->mynode)
- __rcu_process_gp_end(rsp, rnp, rdp);
+ __note_gp_changes(rsp, rnp, rdp);
nocb += rcu_future_gp_cleanup(rsp, rnp);
raw_spin_unlock_irq(&rnp->lock);
cond_resched();
@@ -2254,7 +2229,7 @@ __rcu_process_callbacks(struct rcu_state *rsp)
WARN_ON_ONCE(rdp->beenonline == 0);
/* Handle the end of a grace period that some other CPU ended. */
- rcu_process_gp_end(rsp, rdp);
+ note_gp_changes(rsp, rdp);
/* Update RCU state based on any recent quiescent states. */
rcu_check_quiescent_state(rsp, rdp);
@@ -2340,7 +2315,7 @@ static void __call_rcu_core(struct rcu_state *rsp, struct rcu_data *rdp,
if (unlikely(rdp->qlen > rdp->qlen_last_fqs_check + qhimark)) {
/* Are we ignoring a completed grace period? */
- rcu_process_gp_end(rsp, rdp);
+ note_gp_changes(rsp, rdp);
check_for_new_grace_period(rsp, rdp);
/* Start a new grace period if one not already started. */
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index e6cf7e5..69af628 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -1627,7 +1627,7 @@ static bool rcu_try_advance_all_cbs(void)
*/
if (rdp->completed != rnp->completed &&
rdp->nxttail[RCU_DONE_TAIL] != rdp->nxttail[RCU_NEXT_TAIL])
- rcu_process_gp_end(rsp, rdp);
+ note_gp_changes(rsp, rdp);
if (cpu_has_callbacks_ready_to_invoke(rdp))
cbs_ready = true;
--
1.8.1.5
On Fri, Apr 12, 2013 at 04:32:11PM -0700, Paul E. McKenney wrote:
> This series takes advantage of callback numbering to simplify RCU's
> grace-period machinery, in some cases also reducing the number of
> lock acquisitions (though the resulting change in performance is not
> perceptible). The individual patches are as follows:
>
> 1. Move code to make way for the code-combining in later patches.
> This commit makes no changes, just moves code.
>
> 2. Make __note_new_gpnum() also check for the ends of prior grace
> periods, thus eliminating the earlier possibility of a given
> CPU becoming aware of the start of the next grace period before
> becoming aware of the end of the previous grace period. Yes,
> the code did handle this correctly, but now it doesn't need to.
> More important, now I don't need to think about how it handles
> this correctly.
>
> 3. Rename note_new_gpnum() to note_gp_changes() in preparation for
> later merge of rcu_process_gp_end() into this function.
>
> 4. Change calls to rcu_process_gp_end() to instead call
> note_gp_changes(), and also remove the now-used rcu_process_gp_end().
>
> 5. Remove duplicate code by merging __rcu_process_gp_end() into
> __note_gp_changes().
>
> 6. Eliminate now-redundant call to check_for_new_grace_period(). This
> leaves only a single caller, so inline check_for_new_grace_period().
>
> 7. Given that rcu_start_gp_per_cpu() is a trivial wrapper function
> with only one caller, inline it into its sole remaining call site.
>
> 8. Eliminate now-redundant call to note_gp_changes().
For all 8:
Reviewed-by: Josh Triplett <[email protected]>