Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751515AbXAHNPL (ORCPT ); Mon, 8 Jan 2007 08:15:11 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751523AbXAHNPL (ORCPT ); Mon, 8 Jan 2007 08:15:11 -0500 Received: from thunk.org ([69.25.196.29]:48941 "EHLO thunker.thunk.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751515AbXAHNPI (ORCPT ); Mon, 8 Jan 2007 08:15:08 -0500 Date: Mon, 8 Jan 2007 07:58:19 -0500 From: Theodore Tso To: Suparna Bhattacharya Cc: Andrew Morton , Willy Tarreau , Linus Torvalds , "H. Peter Anvin" , git@vger.kernel.org, nigel@nigel.suspend2.net, "J.H." , Randy Dunlap , Pavel Machek , kernel list , webmaster@kernel.org, "linux-ext4@vger.kernel.org" Subject: Re: How git affects kernel.org performance Message-ID: <20070108125819.GA32756@thunk.org> Mail-Followup-To: Theodore Tso , Suparna Bhattacharya , Andrew Morton , Willy Tarreau , Linus Torvalds , "H. Peter Anvin" , git@vger.kernel.org, nigel@nigel.suspend2.net, "J.H." , Randy Dunlap , Pavel Machek , kernel list , webmaster@kernel.org, "linux-ext4@vger.kernel.org" References: <1166297434.26330.34.camel@localhost.localdomain> <1166304080.13548.8.camel@nigel.suspend2.net> <459152B1.9040106@zytor.com> <1168140954.2153.1.camel@nigel.suspend2.net> <45A08269.4050504@zytor.com> <45A083F2.5000000@zytor.com> <20070107085526.GR24090@1wt.eu> <20070107011542.3496bc76.akpm@osdl.org> <20070108030555.GA7289@in.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070108030555.GA7289@in.ibm.com> User-Agent: Mutt/1.5.12-2006-07-14 X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: tytso@thunk.org X-SA-Exim-Scanned: No (on thunker.thunk.org); SAEximRunCond expanded to false Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1859 Lines: 36 On Mon, Jan 08, 2007 at 08:35:55AM +0530, Suparna Bhattacharya wrote: > > Yeah, slowly-growing directories will get splattered all over the disk. > > > > Possible short-term fixes would be to just allocate up to (say) eight > > blocks when we grow a directory by one block. Or teach the > > directory-growth code to use ext3 reservations. > > > > Longer-term people are talking about things like on-disk rerservations. > > But I expect directories are being forgotten about in all of that. > > By on-disk reservations, do you mean persistent file preallocation ? (that > is explicit preallocation of blocks to a given file) If so, you are > right, we haven't really given any thought to the possibility of directories > needing that feature. The fastest and probably most important thing to add is some readahead smarts to directories --- both to the htree and non-htree cases. If you're using some kind of b-tree structure, such as XFS does for directories, preallocation doesn't help you much. Delayed allocation can save you if your delayed allocator knows how to structure disk blocks so that a btree-traversal is efficient, but I'm guessing the biggest reason why we are losing is because we don't have sufficient readahead. This also has the advantage that it will help without needing to doing a backup/restore to improve layout. Allocating some number of empty blocks when we grow the directory would be a quick hack that I'd probably do as a 2nd priority. It won't help pre-existing directories, but combined with readahead logic, should help us out greatly in the non-btree case. - Ted - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/