Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753221AbbEEWNi (ORCPT ); Tue, 5 May 2015 18:13:38 -0400 Received: from cantor2.suse.de ([195.135.220.15]:56134 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751677AbbEEWNe (ORCPT ); Tue, 5 May 2015 18:13:34 -0400 Date: Tue, 5 May 2015 23:13:29 +0100 From: Mel Gorman To: Andrew Morton Cc: Waiman Long , Nathan Zimmer , Dave Hansen , Scott Norton , Daniel J Blueman , Linux-MM , LKML Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4 Message-ID: <20150505221329.GE2462@suse.de> References: <1430231830-7702-1-git-send-email-mgorman@suse.de> <554030D1.8080509@hp.com> <5543F802.9090504@hp.com> <554415B1.2050702@hp.com> <20150504143046.9404c572486caf71bdef0676@linux-foundation.org> <20150505104514.GC2462@suse.de> <20150505130255.49ff76bbf0a3b32d884ab2ce@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20150505130255.49ff76bbf0a3b32d884ab2ce@linux-foundation.org> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2887 Lines: 64 On Tue, May 05, 2015 at 01:02:55PM -0700, Andrew Morton wrote: > On Tue, 5 May 2015 11:45:14 +0100 Mel Gorman wrote: > > > On Mon, May 04, 2015 at 02:30:46PM -0700, Andrew Morton wrote: > > > > Before the patch, the boot time from elilo prompt to ssh login was 694s. > > > > After the patch, the boot up time was 346s, a saving of 348s (about 50%). > > > > > > Having to guesstimate the amount of memory which is needed for a > > > successful boot will be painful. Any number we choose will be wrong > > > 99% of the time. > > > > > > If the kswapd threads have started, all we need to do is to wait: take > > > a little nap in the allocator's page==NULL slowpath. > > > > > > I'm not seeing any reason why we can't start kswapd much earlier - > > > right at the start of do_basic_setup()? > > > > It doesn't even have to be kswapd, it just should be a thread pinned to > > a done. The difficulty is that dealing with the system hashes means the > > initialisation has to happen before vfs_caches_init_early() when there is > > no scheduler. > > I bet we can run vfs_caches_init_early() after sched_init(). Might > need a few little fixups. > For the large hashes, that would leave the CMA requirement because allocation sizes can be larger than order-10. Arguably on NUMA, that's a bad idea anyway because it should have been interleaved but it's not something this patchset should change. > > Those allocations could be delayed further but then there is > > the possibility that the allocations would not be contiguous and they'd > > have to rely on CMA to make the attempt. That potentially alters the > > performance of the large system hashes at run time. > > hm, why. If the kswapd threads are running and busily creating free > pages then alloc_pages(order=10) can detect this situation and stall > for a while, waiting for kswapd to create an order-10 page. > In Waiman's case, the OOM happened when kswapd was not necessarily available but that's an implementation detail. I'll look tomorrow at what is required to use dedicated threads to parallelisation the allocation and synchronously wait for those threads to complete. It should be possible to create those earlier than kswapd currently is. It'll take longer to boot but hopefully not so long that it makes the series pointless. > Alternatively, the page allocator can go off and synchronously > initialize some pageframes itself. Keep doing that until the > allocation attempt succeeds. > That was rejected during review of earlier attempts at this feature on the grounds that it impacted allocator fast paths. -- Mel Gorman SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/