Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752140AbbD3QKx (ORCPT ); Thu, 30 Apr 2015 12:10:53 -0400 Received: from numascale.com ([213.162.240.84]:60417 "EHLO numascale.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750831AbbD3QKq (ORCPT ); Thu, 30 Apr 2015 12:10:46 -0400 Date: Fri, 01 May 2015 00:10:27 +0800 From: Daniel J Blueman Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4 To: nzimmer , Mel Gorman Cc: Pekka Enberg , Andrew Morton , Dave Hansen , Waiman Long , Scott Norton , Linux-MM , LKML , "'Steffen Persvold'" Message-Id: <1430410227.8193.0@cpanel21.proisp.no> In-Reply-To: <553FD39C.2070503@sgi.com> X-Mailer: geary/0.8.3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel21.proisp.no X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - numascale.com X-Get-Message-Sender-Via: cpanel21.proisp.no: authenticated_id: daniel@numascale.com X-Source: X-Source-Args: X-Source-Dir: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2645 Lines: 67 On Wed, Apr 29, 2015 at 2:38 AM, nzimmer wrote: > On 04/28/2015 11:06 AM, Pekka Enberg wrote: >> On Tue, Apr 28, 2015 at 5:36 PM, Mel Gorman wrote: >>> Struct page initialisation had been identified as one of the >>> reasons why >>> large machines take a long time to boot. Patches were posted a long >>> time ago >>> to defer initialisation until they were first used. This was >>> rejected on >>> the grounds it should not be necessary to hurt the fast paths. This >>> series >>> reuses much of the work from that time but defers the >>> initialisation of >>> memory to kswapd so that one thread per node initialises memory >>> local to >>> that node. >>> >>> After applying the series and setting the appropriate Kconfig >>> variable I >>> see this in the boot log on a 64G machine >>> >>> [ 7.383764] kswapd 0 initialised deferred memory in 188ms >>> [ 7.404253] kswapd 1 initialised deferred memory in 208ms >>> [ 7.411044] kswapd 3 initialised deferred memory in 216ms >>> [ 7.411551] kswapd 2 initialised deferred memory in 216ms >>> >>> On a 1TB machine, I see >>> >>> [ 8.406511] kswapd 3 initialised deferred memory in 1116ms >>> [ 8.428518] kswapd 1 initialised deferred memory in 1140ms >>> [ 8.435977] kswapd 0 initialised deferred memory in 1148ms >>> [ 8.437416] kswapd 2 initialised deferred memory in 1148ms >>> >>> Once booted the machine appears to work as normal. Boot times were >>> measured >>> from the time shutdown was called until ssh was available again. >>> In the >>> 64G case, the boot time savings are negligible. On the 1TB machine, >>> the >>> savings were 16 seconds. > On an older 8 TB box with lots and lots of cpus the boot time, as > measure from grub to login prompt, the boot time improved from 1484 > seconds to exactly 1000 seconds. > > I have time on 16 TB box tonight and a 12 TB box thursday and will > hopefully have more numbers then. Neat, and a roughly similar picture here. On a 7TB, 1728-core NumaConnect system with 108 NUMA nodes, we're seeing stock 4.0 boot in 7136s. This drops to 2159s, or a 70% reduction with this patchset. Non-temporal PMD init [1] drops this to 1045s. Nathan, what do you guys see with the non-temporal PMD patch [1]? Do add a sfence at the ende label if you manually patch. Thanks! Daniel [1] https://lkml.org/lkml/2015/4/23/350 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/