From: Jeff Moyer <jmoyer@redhat.com>
To: Rogier Wolff <R.E.Wolff@BitWizard.nl>
Cc: linux-kernel@vger.kernel.org
Subject: Re: Slow disks.
References: <20101220141553.GA6088@bitwizard.nl>
Date: Mon, 20 Dec 2010 14:09:01 -0500
In-Reply-To: <20101220141553.GA6088@bitwizard.nl> (Rogier Wolff's message of
	"Mon, 20 Dec 2010 15:15:53 +0100")
Message-ID: <x49bp4gw9bm.fsf@segfault.boston.devel.redhat.com>
User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1485
Lines: 36

Rogier Wolff <R.E.Wolff@BitWizard.nl> writes:

> Hi,
>
> A friend of mine has a server in a datacenter somewhere. His machine
> is not working properly: most of his disks take 10-100 times longer
> to process each IO request than normal. 
>
> iostat -kx 10 output: 
> Device: rrqm/s wrqm/s r/s  w/s  rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sdd     0.30   0.00   0.40 1.20 2.80  1.10  4.88     0.43  271.50 271.44  43.43

You are doing ~4KB I/Os and driving a queue depth of <1.  If you are
seeking all over the disk and mixing reads and writes, you may very well
trigger bad behaviour like this for SATA disks.

To further diagnose the issue, I'd recommend running blktrace on one of
the devices.  If you report results here, could you also include more
information about the hardware, the storage layout (including any dm/md
drivers and file systems involved) and the workload?  Also, please let
us know which I/O scheduler you're using.

> These syptoms started when the system was running 2.6.33, but are
> still present now the system has been upgraded to 2.6.36.

Are you sure it was a kernel change that caused the issue?  In other
words, can you run with the 2.6.32 and confirm that the issue is not
present?

Thanks,
Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/