Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752841AbbHWQBM (ORCPT ); Sun, 23 Aug 2015 12:01:12 -0400 Received: from e39.co.us.ibm.com ([32.97.110.160]:52377 "EHLO e39.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751955AbbHWQBK (ORCPT ); Sun, 23 Aug 2015 12:01:10 -0400 X-Helo: d03dlp02.boulder.ibm.com X-MailFrom: paulmck@linux.vnet.ibm.com X-RcptTo: linux-kernel@vger.kernel.org Date: Sun, 23 Aug 2015 09:01:01 -0700 From: "Paul E. McKenney" To: Ingo Molnar Cc: Frederic Weisbecker , Peter Zijlstra , LKML , Vatika Harlalka , Chris Metcalf , Thomas Gleixner , Preeti U Murthy , Christoph Lameter Subject: Re: [PATCH RESEND] sched/nohz: Affine unpinned timers to housekeepers Message-ID: <20150823160101.GA11078@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <1439516774-4614-1-git-send-email-fweisbec@gmail.com> <20150823054032.GA28133@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150823054032.GA28133@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15082316-0033-0000-0000-000005996D2A Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4923 Lines: 130 On Sun, Aug 23, 2015 at 07:40:32AM +0200, Ingo Molnar wrote: > > * Frederic Weisbecker wrote: > > > From: Vatika Harlalka > > > > The problem addressed in this patch is about affining unpinned timers. > > Adaptive or Full Dynticks CPUs are currently disturbed by unnecessary > > jitter due to firing of such timers on them. > > > > This patch will affine timers to online CPUs which are not full dynticks > > in NOHZ_FULL configured systems. It should not introduce overhead in > > nohz full off case due to static keys. > > > > Reviewed-by: Preeti U Murthy > > Signed-off by: Vatika Harlalka > > Cc: Ingo Molnar > > Cc: Peter Zijlstra > > Cc: Christoph Lameter > > Cc: Thomas Gleixner > > Cc: Paul E. McKenney > > Cc: Chris Metcalf > > Signed-off-by: Frederic Weisbecker > > --- > > include/linux/tick.h | 9 ++++++++- > > kernel/sched/core.c | 7 +++++-- > > 2 files changed, 13 insertions(+), 3 deletions(-) > > > > diff --git a/include/linux/tick.h b/include/linux/tick.h > > index 3741ba1..51e6493 100644 > > --- a/include/linux/tick.h > > +++ b/include/linux/tick.h > > @@ -143,13 +143,20 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) > > if (tick_nohz_full_enabled()) > > cpumask_or(mask, mask, tick_nohz_full_mask); > > } > > - > > +static inline int housekeeping_any_cpu(void) > > +{ > > + return cpumask_any_and(housekeeping_mask, cpu_online_mask); > > +} > > extern void __tick_nohz_full_check(void); > > extern void tick_nohz_full_kick(void); > > extern void tick_nohz_full_kick_cpu(int cpu); > > extern void tick_nohz_full_kick_all(void); > > extern void __tick_nohz_task_switch(struct task_struct *tsk); > > #else > > +static inline int housekeeping_any_cpu(void) > > +{ > > + return smp_processor_id(); > > +} > > static inline bool tick_nohz_full_enabled(void) { return false; } > > static inline bool tick_nohz_full_cpu(int cpu) { return false; } > > static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { } > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c > > index 9917c96..4fd42e4 100644 > > --- a/kernel/sched/core.c > > +++ b/kernel/sched/core.c > > @@ -623,18 +623,21 @@ int get_nohz_timer_target(void) > > int i, cpu = smp_processor_id(); > > struct sched_domain *sd; > > > > - if (!idle_cpu(cpu)) > > + if (!idle_cpu(cpu) && is_housekeeping_cpu(cpu)) > > return cpu; > > > > rcu_read_lock(); > > for_each_domain(cpu, sd) { > > for_each_cpu(i, sched_domain_span(sd)) { > > - if (!idle_cpu(i)) { > > + if (!idle_cpu(i) && is_housekeeping_cpu(cpu)) { > > cpu = i; > > goto unlock; > > } > > } > > } > > + > > + if (!is_housekeeping_cpu(cpu)) > > + cpu = housekeeping_any_cpu(); > > unlock: > > rcu_read_unlock(); > > return cpu; > > So I almost applied this yesterday, but had the following question: what ensures > that housekeeping_mask isn't empty? If it's empty then housekeeping_any_cpu() > returns cpumask_any_and() of an empty cpumask - which returns an out of range > index AFAICS - which will crash and burn in: > > kernel/time/hrtimer.c: return &per_cpu(hrtimer_bases, get_nohz_timer_target()); > kernel/time/timer.c: return per_cpu_ptr(&tvec_bases, get_nohz_timer_target()); > > housekeeping_mask itself is derived from tick_nohz_full_mask (it's the inverse of > it in essence), and tick_nohz_full_mask is set via two methods, either via a boot > parameter: > > if (cpulist_parse(str, tick_nohz_full_mask) < 0) { > > in tick_nohz_full_setup(). What ensures here that tick_nohz_full_mask is not > completely full - making housekeeping_mask empty? > > The other method is via CONFIG_NO_HZ_FULL_ALL: > > cpumask_setall(tick_nohz_full_mask); > > here it's fully set - triggering the bug I'm worried about. So what am I missing, > what prevents CONFIG_NO_HZ_FULL_ALL from crashing? The boot CPU is excluded from tick_nohz_full_mask in tick_nohz_init(), which is called from tick_init() which is called from start_kernel() shortly after rcu_init(): cpu = smp_processor_id(); if (cpumask_test_cpu(cpu, tick_nohz_full_mask)) { pr_warning("NO_HZ: Clearing %d from nohz_full range for timekeeping\n", cpu); cpumask_clear_cpu(cpu, tick_nohz_full_mask); } This happens after the call to tick_nohz_init_all() that does the cpumask_setall() that you called out above. Or is a recent patch that I missed changing this? Thanx, Paul -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/