Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1946106Ab3FUVTV (ORCPT ); Fri, 21 Jun 2013 17:19:21 -0400 Received: from relay2.sgi.com ([192.48.179.30]:37228 "EHLO relay.sgi.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1946062Ab3FUVTT (ORCPT ); Fri, 21 Jun 2013 17:19:19 -0400 Message-ID: <51C4C356.3040602@sgi.com> Date: Fri, 21 Jun 2013 14:19:18 -0700 From: Mike Travis User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Yinghai Lu CC: Nathan Zimmer , Robin Holt , Rob Landley , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Andrew Morton , Greg Kroah-Hartman , the arch/x86 maintainers , linux-doc@vger.kernel.org, Linux Kernel Mailing List Subject: Re: [RFC 0/2] Delay initializing of large sections of memory References: <1371831934-156971-1-git-send-email-nzimmer@sgi.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1634 Lines: 41 On 6/21/2013 11:36 AM, Yinghai Lu wrote: > On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer wrote: >> This rfc patch set delays initializing large sections of memory until we have >> started cpus. This has the effect of reducing startup times on large memory >> systems. On 16TB it can take over an hour to boot and most of that time >> is spent initializing memory. > > One hour on system with 16T ram? BIOS or OS? The BIOS is about 20 minutes *before* the 1+ hour. (When we started this UV project way back when, 8TB took over 4 hours before we threw in the towel.) > > I use wall clock to check bootime on one system with 3T and 16 pcie cards, > Linus only takes about 3m and 30 seconds from bootloader. I can send some stats on where various delays are but most of it was in memory initialization. On average UV nodes carry 128 or 256G per node, so 12 nodes would take about 3 or 4 minutes, perhaps more. > > wonder if you boot delay is with so many cpu get onlined in serialized mode. Nope. If that was the case, delaying memory but initializing all the cpus would not affect the time. > > so can you try boot your system with "maxcpus=128" to get the boot time with > wall clock ? We could try if you really think it will provide any useful info. Not sure exactly how having memory on nodes with no active cpus will react. > > Thanks > > Yinghai > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/