Date: Tue, 7 Apr 2015 19:29:23 -0300
From: Marcelo Tosatti <mtosatti@redhat.com>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: linux-kernel@vger.kernel.org, Rik van Riel <riel@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Viresh Kumar <viresh.kumar@linaro.org>,
        Luiz Capitulino <lcapitulino@redhat.com>
Subject: Re: kernel/timer: avoid spurious ksoftirqd wakeups
Message-ID: <20150407222922.GA2494@amt.cnet>
References: <20150402014455.GA25970@amt.cnet>
 <20150402145938.GA10357@lerouge>
 <20150402210809.GC11139@amt.cnet>
 <20150406233413.GB852@lerouge>
 <20150406235126.GA6023@amt.cnet>
 <20150407201720.GA6143@lerouge>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20150407201720.GA6143@lerouge>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 4295
Lines: 88

On Tue, Apr 07, 2015 at 10:17:23PM +0200, Frederic Weisbecker wrote:
> On Mon, Apr 06, 2015 at 08:51:26PM -0300, Marcelo Tosatti wrote:
> > On Tue, Apr 07, 2015 at 01:34:15AM +0200, Frederic Weisbecker wrote:
> > > Yeah, it would be nice to make sure that the cause of these softirqs isn't
> > > mistakenly ignored. 
> > > And also I want to be sure we really understand what we
> > > are doing, which is not the case right now as we don't know what is causing
> > > this expired timer.
> > 
> > What is the interrupt that is the cause for tick_nohz_stop_sched_tick,
> > you mean? 
> > 
> >        <...>-45815 [015] d...2.. 25722056692012 (+961446): kvm_exit: reason EXTERNAL_INTERRUPT rip 0x7f5e448479d0 info 0 800000ef
> >        <...>-45815 [015] d..h1.. 25722056692844 (+832): apic_timer_fn<-__run_hrtimer
> >        <...>-45815 [015] d...1.. 25722056695442 (+2598): raise_softirq_irqoff <-tick_nohz_stop_sched_tick
> > 
> > Emulation of guest APIC timer by hrtimer (apic_timer_fn).
> 
> Nope, I meant what is the root cause of the softirq.
> But lets continue on that below:
> 
> > > Sure, but why is it waking up exactly?
> > 
> > Because there is a bug (the patch is trying to fix the bug by 
> > raising timer softirq only when timer softirq handler has any 
> > work to do).
> > 
> > The only timers pending in the timer list are deferred ones
> > from vmstat_update:
> > 
> > ksoftirqd/15-265   [015] ....111 25722056709372 (+7098): softirq_entry: vec=1 [action=TIMER]
> > ksoftirqd/15-265   [015] .....11 25722056709964 (+592): run_timer_softirq <-do_current_softirqs
> > ksoftirqd/15-265   [015] ....111 25722056714034 (+4070): timer_expire_entry: timer=ffff88082f6f14a0 function=delayed_work_timer_fn now=4480299175
> > ksoftirqd/15-265   [015] ....112 25722056715738 (+1704):
> > workqueue_queue_work: work struct=ffff88082f6f1480 function=vmstat_update workqueue=ffff88041f408000 req_cpu=5120 cpu=15
> > ksoftirqd/15-265   [015] ....112 25722056716304 (+566): workqueue_activate_work: work struct ffff88082f6f1480
> > ksoftirqd/15-265   [015] ....111 25722056719052 (+2748): timer_expire_exit: timer=ffff88082f6f14a0
> > ksoftirqd/15-265   [015] ....111 25722056719384 (+332): softirq_exit: vec=1 [action=TIMER]
> > 
> > Which should only be processed once there are actual add_timer timers
> > being fired (there are no such add_timer timers on this CPU).
> > 
> > Does that make sense?
> 
> So the source of these softirqs is those deffered timers? But defferable timers
> are only defferable in idle-nohz mode, not full-nohz. They are actually deffered
> in practice in full-nohz but it's a bug :o)  (which I need to fix).
> 
> Still, I don't think this is the source of the softirqs since your patch fixes
> the issue of non-timers triggering softirqs.
> 
> So here is the issue: something that is not a "struct timer_list" is causing the
> expiry time of the next tick to be in the past or now. See tick_nohz_stop_sched_tick(),
> the softirq is triggered when delta_jiffies < 1 

delta_jiffies = NEXT_TIMER_MAX_DELTA.

tick_nohz_stop_sched_tick:     delta_jiffies: 1073741823 rcu_delta_jiffies: 18446744073709551615 tick_stopped: 1

> or when the timer fails to be reprogrammed
> because it has already expired.

Right, missed that. I'll ask Luiz to gather info on why its 
failing.

> 
> What can cause this expiry time to be now or in the past? Well for that we need to
> check everything that is used to evaluate the next tick:
> 
> 1) struct timer_list Timers
> 2) low-res hrtimers
> 3) scheduler_tick_max_deferment
> 4) timekeeping_max_deferment
> 5) (rcu|arch|irq_work)_needs_tick()
> 6) maybe something else I'm missing
> 
> Your patch has reduced the softirq to only be triggered in case 1) and it works
> for you. This means the spurious softirqs that you saw were caused by 2,3,4,5 or 6.
> I want to know which one and why because I need to understand exactly which event
> is going to not trigger a softirq anymore after this patch. We want know that to 
> ensure there is no side effect after your patch.
> 
> Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/