Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753150AbbDFXel (ORCPT ); Mon, 6 Apr 2015 19:34:41 -0400 Received: from mail-wg0-f51.google.com ([74.125.82.51]:33782 "EHLO mail-wg0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752840AbbDFXeW (ORCPT ); Mon, 6 Apr 2015 19:34:22 -0400 Date: Tue, 7 Apr 2015 01:34:15 +0200 From: Frederic Weisbecker To: Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, Rik van Riel , Thomas Gleixner , "Paul E. McKenney" , Viresh Kumar Subject: Re: kernel/timer: avoid spurious ksoftirqd wakeups Message-ID: <20150406233413.GB852@lerouge> References: <20150402014455.GA25970@amt.cnet> <20150402145938.GA10357@lerouge> <20150402210809.GC11139@amt.cnet> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150402210809.GC11139@amt.cnet> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4923 Lines: 115 On Thu, Apr 02, 2015 at 06:08:09PM -0300, Marcelo Tosatti wrote: > On Thu, Apr 02, 2015 at 04:59:40PM +0200, Frederic Weisbecker wrote: > > On Wed, Apr 01, 2015 at 10:44:55PM -0300, Marcelo Tosatti wrote: > > > > > > It is only necessary to raise timer softirq > > > in case there are active timers or irq work > > > to do. > > > > > > Limit the ksoftirqd wakeup to those cases. > > > > > > Fixes a latency spike with isolated CPUs and > > > nohz full mode. > > > > > > Reported-and-tested-by: Luiz Capitulino > > > Signed-off-by: Marcelo Tosatti > > > > > > diff --git a/include/linux/timer.h b/include/linux/timer.h > > > index 8c5a197..0c065f9 100644 > > > --- a/include/linux/timer.h > > > +++ b/include/linux/timer.h > > > @@ -192,7 +192,7 @@ extern void set_timer_slack(struct timer_list *time, int slack_hz); > > > * locks the timer base and does the comparison against the given > > > * jiffie. > > > */ > > > -extern unsigned long get_next_timer_interrupt(unsigned long now); > > > +extern unsigned long get_next_timer_interrupt(unsigned long now, bool *raise_softirq); > > > > > > /* > > > * Timer-statistics info: > > > diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c > > > index a4c4eda..615e276 100644 > > > --- a/kernel/time/tick-sched.c > > > +++ b/kernel/time/tick-sched.c > > > @@ -568,6 +568,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, > > > unsigned long rcu_delta_jiffies; > > > struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); > > > u64 time_delta; > > > + bool raise_softirq; > > > > > > time_delta = timekeeping_max_deferment(); > > > > > > @@ -582,9 +583,11 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, > > > arch_needs_cpu() || irq_work_needs_cpu()) { > > > next_jiffies = last_jiffies + 1; > > > delta_jiffies = 1; > > > + raise_softirq = true; > > > > I believe that irq_work doesn't need the softirq. > > Can drop that, right. > > > It needs a tick only in order to call > > irq_work_tick(). And I believe this is the same for RCU which needs a call to > > rcu_check_callbacks(), but it might need something else that the softirq does > > (but this is the timer softirq, not the rcu one). > > > > > } else { > > > /* Get the next timer wheel timer */ > > > - next_jiffies = get_next_timer_interrupt(last_jiffies); > > > + next_jiffies = get_next_timer_interrupt(last_jiffies, > > > + &raise_softirq); > > > delta_jiffies = next_jiffies - last_jiffies; > > > if (rcu_delta_jiffies < delta_jiffies) { > > > next_jiffies = last_jiffies + rcu_delta_jiffies; > > > @@ -703,7 +706,8 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, > > > */ > > > tick_do_update_jiffies64(ktime_get()); > > > } > > > - raise_softirq_irqoff(TIMER_SOFTIRQ); > > > + if (raise_softirq) > > > + raise_softirq_irqoff(TIMER_SOFTIRQ); > > > out: > > > ts->next_jiffies = next_jiffies; > > > ts->last_jiffies = last_jiffies; > > > > Lets look at the things outside the pending timer list that can end up failing > > to program the timer because it is in the past already: > > Is this an attempt to find possible regressions introduced > by this change ? Yeah, it would be nice to make sure that the cause of these softirqs isn't mistakenly ignored. And also I want to be sure we really understand what we are doing, which is not the case right now as we don't know what is causing this expired timer. > > > _ timekeeping_max_deferment(): I doubt, the value is pretty high > > _ scheduler_tick_max_deferment(); it's one second long, way enough to never be in > > the past by the time we program the clock > > _ RCU, irq_work, arch: may be, if the last jiffies update is far enough. But apparently > > the problem is elsewhere since you keep the softirq for these and your patch solves your > > problem. > > _ In case hrtimer runs in low-res mode and the next hrtimer is very close, or even in the past > > already, you may run into such issue. And hrtimer doesn't need the timer softirq, at least not > > to run the callbacks. It needs it only if hrtimer is switching to high-res mode, I think it's > > a rare event. > > > > Now it would be nice to identify the issue we are facing here. Are you running in hrtimer low-res > > mode? > > The issue is a latency spike due to ksoftirqd waking up to > process pending timers, processing two deferred timers, > but no non-deferred timers. > > hrtimer is not in low-res mode. > > The issue is ksoftirqd waking up in the first place. Sure, but why is it waking up exactly? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/