Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753511Ab0LWRrq (ORCPT ); Thu, 23 Dec 2010 12:47:46 -0500 Received: from mx1.redhat.com ([209.132.183.28]:17663 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752862Ab0LWRro (ORCPT ); Thu, 23 Dec 2010 12:47:44 -0500 From: Jeff Moyer To: Rogier Wolff Cc: Greg Freemyer , Bruno =?utf-8?Q?Pr=C3=A9mont?= , linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org Subject: Re: Slow disks. References: <20101220141553.GA6088@bitwizard.nl> <20101220190630.66084e1d@neptune.home> <20101222104306.GB30941@bitwizard.nl> <20101222224416.GE30941@bitwizard.nl> <20101223170109.GA31591@bitwizard.nl> X-PGP-KeyID: 1F78E1B4 X-PGP-CertKey: F6FE 280D 8293 F72C 65FD 5A58 1FF8 A7CA 1F78 E1B4 X-PCLoadLetter: What the f**k does that mean? Date: Thu, 23 Dec 2010 12:47:34 -0500 In-Reply-To: <20101223170109.GA31591@bitwizard.nl> (Rogier Wolff's message of "Thu, 23 Dec 2010 18:01:09 +0100") Message-ID: User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3058 Lines: 74 Rogier Wolff writes: > On Thu, Dec 23, 2010 at 09:40:54AM -0500, Jeff Moyer wrote: >> > In my performance calculations, 10ms average seek (should be around >> > 7), 4ms average rotational latency for a total of 14ms. This would >> > degrade for read-modify-write to 10+4+8 = 22ms. Still 10 times better >> > than what we observe: service times on the order of 200-300ms. >> >> I didn't say it would account for all of your degradation, just that it >> could affect performance. I'm sorry if I wasn't clear on that. > > We can live with a "2x performance degradation" due to stupid > configuration. But not with the 10x -30x that we're seeing now. Wow. I'm not willing to give up any performance due to misconfiguration! >> > > md1 : active raid5 sda2[0] sdd2[3](S) sdb2[1] sdc2[4] >> >> > 39067648 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] >> >> > [UUU] >> >> >> >> A 512KB raid5 chunk with 4KB I/Os? That is a recipe for inefficiency. >> >> Again, blktrace data would be helpful. >> > >> > Where did you get the 4kb IOs from? You mean from the iostat -x >> > output? >> >> Yes, since that's all I have to go on at the moment. >> >> > The system/filesystem decided to do those small IOs. With the >> > throughput we're getting on the filesystem, it better not try to write >> > larger chuncks... >> >> Your logic is a bit flawed, for so many reasons I'm not even going to >> try to enumerate them here. Anyway, I'll continue to sound like a >> broken record and ask for blktrace data. > > Here it is. > > http://prive.bitwizard.nl/blktrace.log > > I can't read those yet... Manual is unclear. OK, I should have made it clear that I wanted the binary logs. No matter, we'll work with what you've sent. > My friend confessed to me today that he determined the "optimal" RAID > block size with the exact same test as I had done, and reached the > same conclusion. So that explains his raid blocksize of 512k. > > The system is a mailserver running on a raid on three of the disks. > most of the IOs are generated by the mail server software through the > FS driver, and the raid system. It's not that we're running a database > that inherently requires 4k IOs. Apparently what the > system needs are those small IOs. The log shows a lot of write barriers: 8,32 0 1183 169.033279975 778 A WBS 481958 + 2 <- (8,34) 8 ^^^ On pre-2.6.37 kernels, that will fully flush the device queue, which is why you're seeing such a small queue depth. There was also a CFQ patch that sped up fsync performance for small files that landed in .37. I can't remember if you ran with a 2.6.37-rc or not. Have you? It may be in your best interest to give the latest -rc a try and report back. Cheers, Jeff -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/