Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759722Ab3HNLGG (ORCPT ); Wed, 14 Aug 2013 07:06:06 -0400 Received: from mail-ea0-f169.google.com ([209.85.215.169]:60889 "EHLO mail-ea0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752168Ab3HNLGE (ORCPT ); Wed, 14 Aug 2013 07:06:04 -0400 Date: Wed, 14 Aug 2013 13:05:56 +0200 From: Ingo Molnar To: Linus Torvalds Cc: Mike Travis , Nathan Zimmer , Peter Anvin , Linux Kernel Mailing List , linux-mm , Robin Holt , Rob Landley , Daniel J Blueman , Andrew Morton , Greg Kroah-Hartman , Yinghai Lu , Mel Gorman Subject: Re: [RFC v3 0/5] Transparent on-demand struct page initialization embedded in the buddy allocator Message-ID: <20130814110556.GH10849@gmail.com> References: <1375465467-40488-1-git-send-email-nzimmer@sgi.com> <1376344480-156708-1-git-send-email-nzimmer@sgi.com> <520A6DFC.1070201@sgi.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3008 Lines: 69 * Linus Torvalds wrote: > [...] > > Ok, so I don't know all the issues, and in many ways I don't even really > care. You could do it other ways, I don't think this is a big deal. The > part I hate is the runtime hook into the core MM page allocation code, > so I'm just throwing out any random thing that comes to my mind that > could be used to avoid that part. So, my hope was that it's possible to have a single, simple, zero-cost runtime check [zero cost for already initialized pages], because it can be merged into already existing page flag mask checks present here and executed for every freshly allocated page: static inline int check_new_page(struct page *page) { if (unlikely(page_mapcount(page) | (page->mapping != NULL) | (atomic_read(&page->_count) != 0) | (page->flags & PAGE_FLAGS_CHECK_AT_PREP) | (mem_cgroup_bad_page_check(page)))) { bad_page(page); return 1; } return 0; } We already run this for every new page allocated and the initialization check could hide in PAGE_FLAGS_CHECK_AT_PREP in a zero-cost fashion. I'd not do any of the ensure_page_is_initialized() or __expand_page_initialization() complications in this patch-set - each page head represents itself and gets iterated when check_new_page() is done. During regular bootup we'd initialize like before, except we don't set up the page heads but memset() them to zero. With each page head 32 bytes this would mean 8 GB of page head memory to clear per 1 TB - with 16 TB that's 128 GB to clear - that ought to be possible to do rather quickly, perhaps with some smart SMP cross-call approach that makes sure that each memset is done in a node-local fashion. [*] Such an approach should IMO be far smaller and less invasive than the patches presented so far: it should be below 100 lines or so. I don't know why there's such a big difference between the theory I outlined and the invasive patch-set implemented so far in practice, perhaps I'm missing some complication. I was trying to probe that difference, before giving up on the idea and punting back to the async hotplug-ish approach which would obviously work well too. All in one, I think async init just hides the real problem - there's no way memory init should take this long. Thanks, Ingo [*] alternatively maybe the main performance problem is that node-local memory is set up on a remote (boot) node? In that case I'd try to optimize it by migrating the memory init code's current node by using set_cpus_allowed() to live migrate from node to node, tracking the node whose struct page array is being initialized. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/