Date: Tue, 14 Jun 2011 10:13:15 +0200
From: Ingo Molnar <mingo@elte.hu>
To: Shaohua Li <shaohua.li@intel.com>
Cc: paulmck@linux.vnet.ibm.com, lkml <linux-kernel@vger.kernel.org>,
        "Chen, Tim C" <tim.c.chen@intel.com>, "Shi, Alex" <alex.shi@intel.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Peter Zijlstra <a.p.zijlstra@chello.nl>,
        Thomas Gleixner <tglx@linutronix.de>
Subject: Re: rcu: performance regression
Message-ID: <20110614081315.GE29900@elte.hu>
References: <1308029185.15392.147.camel@sli10-conroe>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1308029185.15392.147.camel@sli10-conroe>
User-Agent: Mutt/1.5.20 (2009-08-17)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2157
Lines: 52


* Shaohua Li <shaohua.li@intel.com> wrote:

> Commit a26ac2455ffcf3(rcu: move TREE_RCU from softirq to kthread)
> introduced performance regression. In our AIM7 test, this commit caused
> about 40% regression.

Sigh, this commit is somewhat of a train-wreck.

> The commit runs rcu callbacks in a kthread instead of softirq. We 
> observed high rate of context switch which is caused by this. Out 
> test system has 64 CPUs and HZ is 1000, so we saw more than 64k 
> context switch per second which is caused by the rcu thread.
>
> I also did trace and found when rcy thread is woken up, most time 
> the thread doesn't handle any callbacks actually, it just 
> initializes new gp or end one gp or similar.
>
> From my understanding, the purpose to make rcu runs in kthread is 
> to speed up rcu callbacks run (with help of rtmutex PI), not for 
> end gp and so on, which runs pretty fast actually and doesn't need 
> boost. To verify my findings, I had below debug patch applied. It 
> still handles rcu callbacks in kthread if there is any pending 
> callbacks, but other things are still running in softirq. this 
> completely solved our regression. I thought this can still boost 
> callbacks run. but I'm not expert in the area, so please help.
> 
> Thanks,
> Shaohua
> ---
>  Documentation/filesystems/proc.txt  |    1 +
>  include/linux/interrupt.h           |    1 +
>  include/trace/events/irq.h          |    3 ++-
>  kernel/rcutree.c                    |   23 +++++++++++++++++++----
>  kernel/rcutree.h                    |    1 +
>  kernel/rcutree_plugin.h             |    9 +++++++++
>  kernel/softirq.c                    |    2 +-
>  tools/perf/util/trace-event-parse.c |    1 +
>  8 files changed, 35 insertions(+), 6 deletions(-)

Paul? Unless this patch is the obviously correct solution everyone 
wants to have, the other obviously correct solution is to do the 
revert ...

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/