Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760247AbZAWEJY (ORCPT ); Thu, 22 Jan 2009 23:09:24 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754643AbZAWEJQ (ORCPT ); Thu, 22 Jan 2009 23:09:16 -0500 Received: from ns2.suse.de ([195.135.220.15]:33258 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754547AbZAWEJP (ORCPT ); Thu, 22 Jan 2009 23:09:15 -0500 Date: Fri, 23 Jan 2009 05:09:13 +0100 From: Nick Piggin To: Christoph Lameter Cc: Pekka Enberg , "Zhang, Yanmin" , Lin Ming , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton , Linus Torvalds Subject: Re: [patch] SLQB slab allocator Message-ID: <20090123040913.GG20098@wotan.suse.de> References: <20090114152207.GD25401@wotan.suse.de> <84144f020901140730l747b4e06j41fb8a35daeaf6c8@mail.gmail.com> <20090114155923.GC1616@wotan.suse.de> <20090115061931.GC17810@wotan.suse.de> <20090116034356.GM17810@wotan.suse.de> <20090119061856.GB22584@wotan.suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.9i Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4045 Lines: 89 On Wed, Jan 21, 2009 at 07:13:44PM -0500, Christoph Lameter wrote: > On Mon, 19 Jan 2009, Nick Piggin wrote: > > > > The per cpu queue size in SLUB is limited by the queues only containing > > > objects from the same page. If you have large queues like SLAB/SLQB(?) > > > then this could be an issue. > > > > And it could be a problem in SLUB too. Chances are that several allocations > > will be wrong after every policy switch. I could describe situations in which > > SLUB will allocate with the _wrong_ policy literally 100% of the time. > > No it cannot because in SLUB objects must come from the same page. > Multiple objects in a queue will only ever require a single page and not > multiple like in SLAB. I don't know how that solves the problem. Task with memory policy A allocates an object, which allocates the "fast" page with policy A and allocates an object. Then context switch to task with memory policy B which allocates another object, which is taken from the page allocated with policy A. Right? (OK this doesn't give the wrong policy 100% of the time; I thought there could have been a context switch race during page allocation that would result in 100% incorrect, but anyway it could still be significantly incorrect couldn't it?) > > > That means large amounts of memory are going to be caught in these queues. > > > If its per cpu and one cpu does allocation and the other frees then the > > > first cpu will consume more and more memory from the page allocator > > > whereas the second will build up huge per cpu lists. > > > > Wrong. I said I would allow an option to turn off *periodic trimming*. > > Or just modify the existing tunables or look at making the trimming > > more fine grained etc etc. I won't know until I see a workload where it > > hurts, and I will try to solve it then. > > You are not responding to the issue. If you have queues that contain > objects from multiple pages then every object pointer in these queues can > pin a page although this actually is a free object. I am trying to respond to what you raise. "The" issue I thought you raised above was that SLQB would grow freelists unbounded "the first cpu will consume more and more memory from the page allocator whereas the second will build up huge per cpu lists" And this is wrong. There is another possible issue where every single object on the freelist might come from a different (and otherwise free) page, and thus eg 100 8 byte objects might consume 400K. That's not an invalid concern, but I think it will be quite rare, and the periodic queue trimming should naturally help this because it will cycle out those objects and if new allocations are needed, they will come from new pages which can be packed more densely. > > > It seems that on SMP systems SLQB will actually increase the number of > > > queues since it needs 2 queues per cpu instead of the 1 of SLAB. > > > > I don't know what you mean when you say queues, but SLQB has more > > than 2 queues per CPU. Great. I like them ;) > > This gets better and better. So no response to my asking where the TLB improvement in SLUB helps, or where queueing hurts? You complain about not being able to reproduce Intel's OLTP problem, and yet you won't even _say_ what the problems are for SLQB. Wheras Intel at least puts a lot of effort into running tests and helping to analyse things. > > > SLAB also > > > has resizable queues. > > > > Not significantly because that would require large memory allocations for > > large queues. And there is no code there to do runtime resizing. > > Groan. Please have a look at do_tune_cpucache() in slab.c Cool, I didn't realise it had hooks to do runtime resizing. The more important issue of course is the one of extra cache footprint and metadata in SLAB's scheme. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/