Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757064AbZCCVsz (ORCPT ); Tue, 3 Mar 2009 16:48:55 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754676AbZCCVsq (ORCPT ); Tue, 3 Mar 2009 16:48:46 -0500 Received: from gir.skynet.ie ([193.1.99.77]:34859 "EHLO gir.skynet.ie" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754473AbZCCVsp (ORCPT ); Tue, 3 Mar 2009 16:48:45 -0500 Date: Tue, 3 Mar 2009 21:48:42 +0000 From: Mel Gorman To: Christoph Lameter Cc: Lin Ming , Pekka Enberg , Linux Memory Management List , Rik van Riel , KOSAKI Motohiro , Johannes Weiner , Nick Piggin , Linux Kernel Mailing List , Zhang Yanmin , Peter Zijlstra , Ingo Molnar Subject: Re: [RFC PATCH 00/19] Cleanup and optimise the page allocator V2 Message-ID: <20090303214842.GL10577@csn.ul.ie> References: <1235477835-14500-1-git-send-email-mel@csn.ul.ie> <1235639427.11390.11.camel@minggr> <20090226110336.GC32756@csn.ul.ie> <1235647139.16552.34.camel@penberg-laptop> <20090226112232.GE32756@csn.ul.ie> <1235724283.11610.212.camel@minggr> <20090302112122.GC21145@csn.ul.ie> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17+20080114 (2008-01-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2246 Lines: 41 On Tue, Mar 03, 2009 at 11:31:46AM -0500, Christoph Lameter wrote: > On Mon, 2 Mar 2009, Mel Gorman wrote: > > > Going by the vanilla kernel, a *large* amount of time is spent doing > > high-order allocations. Over 25% of the cost of buffered_rmqueue() is in > > the branch dealing with high-order allocations. Does UDP-U-4K mean that 8K > > pages are required for the packets? That means high-order allocations and > > high contention on the zone-list. That is bad obviously and has implications > > for the SLUB-passthru patch because whether 8K allocations are handled by > > SL*B or the page allocator has a big impact on locking. > > > > Next, a little over 50% of the cost get_page_from_freelist() is being spent > > acquiring the zone spinlock. The implication is that the SL*B allocators > > passing in order-1 allocations to the page allocator are currently going to > > hit scalability problems in a big way. The solution may be to extend the > > per-cpu allocator to handle magazines up to PAGE_ALLOC_COSTLY_ORDER. I'll > > check it out. > > Then we are increasing the number of queues dramatically in the page > allocator. More of a memory sink. Less cache hotness. > It doesn't have to be more queues and networking is doing order-1 allocations based on a quick instrumentation so we might be justified in doing this to avoid contending excessively on the zone lock. Without the patchset, we do a search of the pcp lists for a page of the appropriate migrate type. There is a patch that removes this search at the cost of one cache line per CPU and it works reasonably well. However, if the search was left in place, you can add pages of other orders and just search for those which should be a lot less costly. Yes, the search is unfortunate but you avoid acquiring the zone lock without increasing the size of the per-cpu structure. The search will require cache lines it's probably less than acquiring teh zone lock and going through the whole buddy allocator for order-1 pages. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/