Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754680Ab2E2RPG (ORCPT ); Tue, 29 May 2012 13:15:06 -0400 Received: from mx1.redhat.com ([209.132.183.28]:1026 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754644Ab2E2RPC (ORCPT ); Tue, 29 May 2012 13:15:02 -0400 Date: Tue, 29 May 2012 19:14:07 +0200 From: Andrea Arcangeli To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Hillf Danton , Dan Smith , Linus Torvalds , Andrew Morton , Thomas Gleixner , Ingo Molnar , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" , Lai Jiangshan , Bharata B Rao , Lee Schermerhorn , Rik van Riel , Johannes Weiner , Srivatsa Vaddagiri , Christoph Lameter Subject: Re: [PATCH 35/35] autonuma: page_autonuma Message-ID: <20120529171407.GH21339@redhat.com> References: <1337965359-29725-1-git-send-email-aarcange@redhat.com> <1337965359-29725-36-git-send-email-aarcange@redhat.com> <1338309855.26856.130.camel@twins> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1338309855.26856.130.camel@twins> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3567 Lines: 79 On Tue, May 29, 2012 at 06:44:15PM +0200, Peter Zijlstra wrote: > On Fri, 2012-05-25 at 19:02 +0200, Andrea Arcangeli wrote: > > Move the AutoNUMA per page information from the "struct page" to a > > separate page_autonuma data structure allocated in the memsection > > (with sparsemem) or in the pgdat (with flatmem). > > > > This is done to avoid growing the size of the "struct page" and the > > page_autonuma data is only allocated if the kernel has been booted on > > real NUMA hardware (or if noautonuma is passed as parameter to the > > kernel). > > > > Argh, please fold this change back into the series proper. If you want > to keep it.. as it is its not really an improvement IMO, see below. The whole objective of this patch is to avoid allocating the page_autonuma structures when the kernel is booted on not NUMA hardware. It's not an improvement when booting the kernel on NUMA hardware that's for sure. I didn't merge it with the previous because this was the most experimental recent change, so I wanted bisectability here. When something goes wrong here, the kernel won't boot, so unless you use kvm with gdbstub it's a little tricky to debug (indeed I debugged it with gdbstub, there it's trivial). > > +struct page_autonuma { > > + /* > > + * FIXME: move to pgdat section along with the memcg and allocate > > + * at runtime only in presence of a numa system. > > + */ > > + /* > > + * To modify autonuma_last_nid lockless the architecture, > > + * needs SMP atomic granularity < sizeof(long), not all archs > > + * have that, notably some alpha. Archs without that requires > > + * autonuma_last_nid to be a long. > > + */ > > Looking at arch/alpha/include/asm/xchg.h it looks to have that just > fine, so maybe we simply don't support SMP on those early Alphas that > had that weirdness. I agree we should never risk that. > This makes a shadow page frame of 32 bytes per page, or ~0.8% of memory. > This isn't in fact an improvement. > > The suggestion done by Rik was to have something like a sqrt(nr_pages) > (?) scaled array of such things containing the list_head and page > pointer -- and leave the two nids in the regular page frame. Although I > think you've got to fight the memcg people over that last word in struct > page. > > That places a limit on the amount of pages that can be in migration > concurrently, but also greatly reduces the memory overhead. Yes, however for the last_nid I'd still need it for every page (and if I allocate it dynamic I still first need to find a way to remove the struct page pointer). I thought to add a pointer in the memsection (or maybe to use a vmemmap so that it won't even require a pointer in every memsection). I've to check a few more things before I allow_the autonuma->page translation without a page pointer, notably to verify the boot time allocations points won't just allocate power of two blocks of memory (they shouldn't but I didn't verify). This is clearly a move in the right direction to avoid the memory overhead when not booted on NUMA hardware, and I don't think there's anything fundamental that prevents us remove the page pointer from the page_autonuma structure, and to later experiment with a limited size array of async migration structures. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/