Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756707Ab3DPMGl (ORCPT ); Tue, 16 Apr 2013 08:06:41 -0400 Received: from relay1.sgi.com ([192.48.179.29]:50410 "EHLO relay.sgi.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755121Ab3DPMG2 (ORCPT ); Tue, 16 Apr 2013 08:06:28 -0400 Date: Tue, 16 Apr 2013 07:06:24 -0500 From: Robin Holt To: Ingo Molnar Cc: Robin Holt , Ingo Molnar , Russ Anderson , Shawn Guo , Oleg Nesterov , Andrew Morton , "H. Peter Anvin" , Joe Perches , Lai Jiangshan , Linus Torvalds , Linux Kernel Mailing List , Michel Lespinasse , "Paul E. McKenney" , Paul Mackerras , Peter Zijlstra , "rusty@rustcorp.com.au" , Tejun Heo , the arch/x86 maintainers , Thomas Gleixner Subject: Re: [Patch -v4 1/4] Migrate shutdown/reboot to boot cpu. Message-ID: <20130416120624.GO3658@sgi.com> References: <20130416095827.GJ3672@sgi.com> <20130416113256.GA10452@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130416113256.GA10452@gmail.com> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3756 Lines: 108 On Tue, Apr 16, 2013 at 01:32:56PM +0200, Ingo Molnar wrote: > > * Robin Holt wrote: > > > We recently noticed that reboot of a 1024 cpu machine takes approx 16 > > minutes of just stopping the cpus. The slowdown was tracked to commit > > f96972f. > > > > The current implementation does all the work of hot removing the cpus > > before halting the system. We are switching to just migrating to the > > boot cpu and then continuing with shutdown/reboot. > > > > This also has the effect of not breaking x86's command line parameter for > > specifying the reboot cpu. Note, this code was shamelessly copied from > > arch/x86/kernel/reboot.c with bits removed pertaining to the reboot_cpu > > command line parameter. > > > > Signed-off-by: Robin Holt > > Tested-by: Shawn Guo > > To: Ingo Molnar > > To: Russ Anderson > > To: Oleg Nesterov > > Cc: Andrew Morton > > Cc: "H. Peter Anvin" > > Cc: Lai Jiangshan > > Cc: Linus Torvalds > > Cc: Linux Kernel Mailing List > > Cc: Michel Lespinasse > > Cc: Oleg Nesterov > > Cc: "Paul E. McKenney" > > Cc: Paul Mackerras > > Cc: Peter Zijlstra > > Cc: Robin Holt > > Cc: "rusty@rustcorp.com.au" > > Cc: Tejun Heo > > Cc: the arch/x86 maintainers > > Cc: Thomas Gleixner > > Cc: > > > > --- > > > > Changes since -v1. > > - Set PF_THREAD_BOUND before migrating to eliminate potential race. > > - Modified kernel_power_off to also migrate instead of using > > disable_nonboot_cpus(). > > --- > > kernel/sys.c | 22 +++++++++++++++++++--- > > 1 file changed, 19 insertions(+), 3 deletions(-) > > > > diff --git a/kernel/sys.c b/kernel/sys.c > > index 0da73cf..5ef7aa2 100644 > > --- a/kernel/sys.c > > +++ b/kernel/sys.c > > @@ -357,6 +357,22 @@ int unregister_reboot_notifier(struct notifier_block *nb) > > } > > EXPORT_SYMBOL(unregister_reboot_notifier); > > > > +void migrate_to_reboot_cpu(void) > > It appears to be file-scope, so should be static I guess? Done. > > +{ > > + /* The boot cpu is always logical cpu 0 */ > > + int reboot_cpu_id = 0; > > + > > + /* Make certain the cpu I'm about to reboot on is online */ > > + if (!cpu_online(reboot_cpu_id)) > > + reboot_cpu_id = smp_processor_id(); > > Shouldn't we pick the first online CPU instead, to make it deterministic? Done. reboot_cpu_id = cpumask_first(cpu_online_mask); > Also, does this codepath prevent hotplug from going on in parallel? Not sure. I have not considered hotplug. I will look that over when I am in the office. > ( Plus, the smp_processor_id() is in a preemptible section AFAICS, so it will > throw a warning with preempt debug on. ) > > > + > > + /* Prevent races with other tasks migrating this task. */ > > (I guess the colon can be dropped here, like in the other comments.) Done. > > > + current->flags |= PF_THREAD_BOUND; > > + > > + /* Make certain I only run on the appropriate processor */ > > + set_cpus_allowed_ptr(current, cpumask_of(reboot_cpu_id)); > > +} I will resubmit when I have the hotplug stuff understood and after giving the set some more time for comments. Robin -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/