Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753398Ab0LWRBM (ORCPT ); Thu, 23 Dec 2010 12:01:12 -0500 Received: from dtp.xs4all.nl ([80.101.171.8]:60803 "HELO abra2.bitwizard.nl" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with SMTP id S1752973Ab0LWRBL (ORCPT ); Thu, 23 Dec 2010 12:01:11 -0500 Date: Thu, 23 Dec 2010 18:01:09 +0100 From: Rogier Wolff To: Jeff Moyer Cc: Rogier Wolff , Greg Freemyer , Bruno =?iso-8859-1?Q?Pr=E9mont?= , linux-kernel@vger.kernel.org, linux-ide@vger.kernel.org Subject: Re: Slow disks. Message-ID: <20101223170109.GA31591@bitwizard.nl> References: <20101220141553.GA6088@bitwizard.nl> <20101220190630.66084e1d@neptune.home> <20101222104306.GB30941@bitwizard.nl> <20101222224416.GE30941@bitwizard.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: BitWizard.nl User-Agent: Mutt/1.5.13 (2006-08-11) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3578 Lines: 81 On Thu, Dec 23, 2010 at 09:40:54AM -0500, Jeff Moyer wrote: > > In my performance calculations, 10ms average seek (should be around > > 7), 4ms average rotational latency for a total of 14ms. This would > > degrade for read-modify-write to 10+4+8 = 22ms. Still 10 times better > > than what we observe: service times on the order of 200-300ms. > > I didn't say it would account for all of your degradation, just that it > could affect performance. I'm sorry if I wasn't clear on that. We can live with a "2x performance degradation" due to stupid configuration. But not with the 10x -30x that we're seeing now. > > > md1 : active raid5 sda2[0] sdd2[3](S) sdb2[1] sdc2[4] > >> > 39067648 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] > >> > [UUU] > >> > >> A 512KB raid5 chunk with 4KB I/Os? That is a recipe for inefficiency. > >> Again, blktrace data would be helpful. > > > > Where did you get the 4kb IOs from? You mean from the iostat -x > > output? > > Yes, since that's all I have to go on at the moment. > > > The system/filesystem decided to do those small IOs. With the > > throughput we're getting on the filesystem, it better not try to write > > larger chuncks... > > Your logic is a bit flawed, for so many reasons I'm not even going to > try to enumerate them here. Anyway, I'll continue to sound like a > broken record and ask for blktrace data. Here it is. http://prive.bitwizard.nl/blktrace.log I can't read those yet... Manual is unclear. I'd guess that "D" means "submitted to driver". and "C" means "completed". I very often see a D followed VERY shortly by a C. Also I see more C's than "D"s. Anohter way of looking at it, was to sort on the "ID" field. I would expect each "transaction" to follow similar steps. But many IDs only occur twice, and not the same for each. > > I have benchmarked my own "high bandwidth" raid arrays. I benchmarked > > them with 128k, 256, 512 and 1024k blocksize. I got the best > > throughput (for my benchmark: dd if=/dev/md0 of=/dev/null bs=1024k) > > with 512k blocksize. (and yes that IS a valid benchmark for my > > usage of the array.) > > Sorry, I'm not sure I understand how this is relevant. I thought we > were troubleshooting a problem on someone else's system. Further, the > window into the workload we saw via iostat definitely shows that smaller > I/Os are issued. My friend confessed to me today that he determined the "optimal" RAID block size with the exact same test as I had done, and reached the same conclusion. So that explains his raid blocksize of 512k. The system is a mailserver running on a raid on three of the disks. most of the IOs are generated by the mail server software through the FS driver, and the raid system. It's not that we're running a database that inherently requires 4k IOs. Apparently what the system needs are those small IOs. Roger. -- ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/