by Paul E. McKenney

[permalink] [raw]

Subject: Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

On Fri, Jun 10, 2011 at 03:14:29PM +0800, Lai Jiangshan wrote:
> On 06/09/2011 03:29 AM, Paul E. McKenney wrote:
> > Given some common flag combinations, particularly -Os, gcc will inline
> > rcu_read_unlock_special() despite its being in an unlikely() clause.
> > Use noline to prohibit this misoptimization.
> >
> > In addition, move the second barrier() in __rcu_read_unlock() so that
> > it is not on the common-case code path. This will allow the compiler to
> > generate better code for the common-case path through __rcu_read_unlock().
> >
> > Finally, fix up whitespace in kernel/lockdep.c to keep checkpatch happy.
> >
> > Suggested-by: Linus Torvalds <[email protected]>
> > Signed-off-by: Paul E. McKenney <[email protected]>
> > ---
> > kernel/rcutree_plugin.h | 12 ++++++------
> > 1 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> > index ea2e2fb..40a6db7 100644
> > --- a/kernel/rcutree_plugin.h
> > +++ b/kernel/rcutree_plugin.h
> > @@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
> > * notify RCU core processing or task having blocked during the RCU
> > * read-side critical section.
> > */
> > -static void rcu_read_unlock_special(struct task_struct *t)
> > +static noinline void rcu_read_unlock_special(struct task_struct *t)
> > {
> > int empty;
> > int empty_exp;
> > @@ -387,11 +387,11 @@ void __rcu_read_unlock(void)
> > struct task_struct *t = current;
> >
> > barrier(); /* needed if we ever invoke rcu_read_unlock in rcutree.c */
> > - --t->rcu_read_lock_nesting;
> > - barrier(); /* decrement before load of ->rcu_read_unlock_special */
> > - if (t->rcu_read_lock_nesting == 0 &&
> > - unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> > - rcu_read_unlock_special(t);
> > + if (--t->rcu_read_lock_nesting == 0) {
>
> > + barrier(); /* decr before ->rcu_read_unlock_special load */
>
> Since ACCESS_ONCE() is used for loading ->rcu_read_unlock_special, is the previous
> barrier() still needed?

It doesn't really matter until we can inline __rcu_read_unlock(), but
hopefully that day is coming soon. So...

The concern is for cases where the compiler can see __rcu_read_lock() and
__rcu_read_unlock(). The compiler would then be within its rights to
cancel the increments and decrements of t->rcu_read_lock_nesting against
each other, which could turn a loop containing an RCU read-side critical
section into one big long critical section.

We could do --ACCESS_ONCE(t->rcu_read_lock_nesting), but that generates
lousy code on x86. So, is there a way to make the compiler forget only
about t->rcu_read_lock_nesting rather than about all variables?

Thanx, Paul

> > + if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> > + rcu_read_unlock_special(t);
> > + }
> > #ifdef CONFIG_PROVE_LOCKING
> > WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
> > #endif /* #ifdef CONFIG_PROVE_LOCKING */
>

2011-06-10 19:53:56

by Paul E. McKenney

[permalink] [raw]

Subject: Re: [PATCH tip/core/rcu 01/28] rcu: Simplify curing of load woes

On Fri, Jun 10, 2011 at 04:18:32PM +0200, Peter Zijlstra wrote:
> On Wed, 2011-06-08 at 12:29 -0700, Paul E. McKenney wrote:
> > Make the functions creating the kthreads wake them up. Leverage the
> > fact that the per-node and boost kthreads can run anywhere, thus
> > dispensing with the need to wake them up once the incoming CPU has
> > gone fully online.
>
> Indeed, I failed to notice the node and boost threads weren't bound.

Hey, you did the big fix, so I cannot complain about doing a little
cleanup! ;-)

> > Signed-off-by: Paul E. McKenney <[email protected]>
> > ---
> > kernel/rcutree.c | 65 +++++++++++++++-------------------------------
> > kernel/rcutree_plugin.h | 11 +-------
> > 2 files changed, 22 insertions(+), 54 deletions(-)
> >
> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > index 4cc6a94..36e79d2 100644
> > --- a/kernel/rcutree.c
> > +++ b/kernel/rcutree.c
> > @@ -1634,6 +1634,20 @@ static int rcu_cpu_kthread(void *arg)
> > * to manipulate rcu_cpu_kthread_task. There might be another CPU
> > * attempting to access it during boot, but the locking in kthread_bind()
> > * will enforce sufficient ordering.
> > + *
> > + * Please note that we cannot simply refuse to wake up the per-CPU
> > + * kthread because kthreads are created in TASK_UNINTERRUPTIBLE state,
> > + * which can result in softlockup complaints if the task ends up being
> > + * idle for more than a couple of minutes.
> > + *
> > + * However, please note also that we cannot bind the per-CPU kthread to its
> > + * CPU until that CPU is fully online. We also cannot wait until the
> > + * CPU is fully online before we create its per-CPU kthread, as this would
> > + * deadlock the system when CPU notifiers tried waiting for grace
> > + * periods. So we bind the per-CPU kthread to its CPU only if the CPU
> > + * is online. If its CPU is not yet fully online, then the code in
> > + * rcu_cpu_kthread() will wait until it is fully online, and then do
> > + * the binding.
> > */
> > static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
> > {
> > @@ -1646,12 +1660,14 @@ static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
> > t = kthread_create(rcu_cpu_kthread, (void *)(long)cpu, "rcuc%d", cpu);
> > if (IS_ERR(t))
> > return PTR_ERR(t);
> > - kthread_bind(t, cpu);
> > + if (cpu_online(cpu))
> > + kthread_bind(t, cpu);
> > per_cpu(rcu_cpu_kthread_cpu, cpu) = cpu;
> > WARN_ON_ONCE(per_cpu(rcu_cpu_kthread_task, cpu) != NULL);
> > - per_cpu(rcu_cpu_kthread_task, cpu) = t;
> > sp.sched_priority = RCU_KTHREAD_PRIO;
> > sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
> > + per_cpu(rcu_cpu_kthread_task, cpu) = t;
> > + wake_up_process(t); /* Get to TASK_INTERRUPTIBLE quickly. */
> > return 0;
> > }
>
> I'm not quite seeing how this is working though, I cannot find any code
> in rcu_cpu_kthread() that sets the thread affinity (not a hunk in this
> patch that adds it).

This happens in rcu_cpu_kthread_should_stop(), which is called from
rcu_cpu_kthread() before it does any real work.

Here it is:

static int rcu_cpu_kthread_should_stop(int cpu)
{
while (cpu_is_offline(cpu) ||
!cpumask_equal(&current->cpus_allowed, cpumask_of(cpu)) ||
smp_processor_id() != cpu) {
if (kthread_should_stop())
return 1;
per_cpu(rcu_cpu_kthread_status, cpu) = RCU_KTHREAD_OFFCPU;
per_cpu(rcu_cpu_kthread_cpu, cpu) = raw_smp_processor_id();
local_bh_enable();
schedule_timeout_uninterruptible(1);
if (!cpumask_equal(&current->cpus_allowed, cpumask_of(cpu)))
set_cpus_allowed_ptr(current, cpumask_of(cpu));
local_bh_disable();
}
per_cpu(rcu_cpu_kthread_cpu, cpu) = cpu;
return 0;
}

Thoughts?

Thanx, Paul