Hello;
I am running aio-stress on the most recent kernels and have
found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
performance of buffered random reads is poor compared to the
buffered random writes:
2.6.8 2.6.9-rc2 2.6.9-rc2-mm4
--------------------------------------------
random write 35.66 MB/s 34.80 MB/s 29.89 MB/s
random read 7.69 MB/s 7.50 MB/s 7.68 MB/s
** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
This shows up on our 4CPU host as well. (striped AACRAID.4G
RAM. 8G File):
2.6.9-rc2 2.6.9-rc2-mm4 2.6.9-rc2-mm1
-------------------------------------------
random write 31.36 MB/s 18.92 MB/s 18.97 MB/s
random read 11.13 MB/s 9.74 MB/s 11.05 MB/s
There seems to be an issue with the reads. Usually, reads
should be at least as fast as writes of the same type.
Also, there seems to be a substantial drop-off in the performance
of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
40% on 4CPU)
Regards;
Judith Lebzelter
OSDL
On Wed, Sep 29, 2004 at 04:29:08PM -0700, Judith Lebzelter wrote:
> There seems to be an issue with the reads. Usually, reads
> should be at least as fast as writes of the same type.
> Also, there seems to be a substantial drop-off in the performance
> of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> 40% on 4CPU)
Okay, is it cpu time or idle/iowait? If it's cpu time, where do
profiles show it appears?
-- wli
Judith Lebzelter <[email protected]> wrote:
>
> Hello;
>
> I am running aio-stress on the most recent kernels and have
> found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
> performance of buffered random reads is poor compared to the
> buffered random writes:
>
> 2.6.8 2.6.9-rc2 2.6.9-rc2-mm4
> --------------------------------------------
> random write 35.66 MB/s 34.80 MB/s 29.89 MB/s
> random read 7.69 MB/s 7.50 MB/s 7.68 MB/s
>
> ** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
>
>
> This shows up on our 4CPU host as well. (striped AACRAID.4G
> RAM. 8G File):
> 2.6.9-rc2 2.6.9-rc2-mm4 2.6.9-rc2-mm1
> -------------------------------------------
> random write 31.36 MB/s 18.92 MB/s 18.97 MB/s
> random read 11.13 MB/s 9.74 MB/s 11.05 MB/s
>
>
> There seems to be an issue with the reads. Usually, reads
> should be at least as fast as writes of the same type.
>
> Also, there seems to be a substantial drop-off in the performance
> of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> 40% on 4CPU)
>
Well one would expect writes to be much faster than reads because writes
usually do not involve performing physical I/O, and when pagecache
writeback finally happens it has vastly more data to work with and hence
can schedule I/O more efficiently.
Unless you are using O_SYNC or fsync(), in which case ignore the above.
The regression within random write performance is unexpected. Can you
please provide a URL to the current version of the test tool, and a
description of how you are invoking it? What sort of I/O system, what
filesystem, etc.
Thanks.
On Wed, 29 Sep 2004, Andrew Morton wrote:
> Judith Lebzelter <[email protected]> wrote:
> >
> > Hello;
> >
> > I am running aio-stress on the most recent kernels and have
> > found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
> > performance of buffered random reads is poor compared to the
> > buffered random writes:
> >
> > 2.6.8 2.6.9-rc2 2.6.9-rc2-mm4
> > --------------------------------------------
> > random write 35.66 MB/s 34.80 MB/s 29.89 MB/s
> > random read 7.69 MB/s 7.50 MB/s 7.68 MB/s
> >
> > ** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
> >
> >
> > This shows up on our 4CPU host as well. (striped AACRAID.4G
> > RAM. 8G File):
> > 2.6.9-rc2 2.6.9-rc2-mm4 2.6.9-rc2-mm1
> > -------------------------------------------
> > random write 31.36 MB/s 18.92 MB/s 18.97 MB/s
> > random read 11.13 MB/s 9.74 MB/s 11.05 MB/s
> >
> >
> > There seems to be an issue with the reads. Usually, reads
> > should be at least as fast as writes of the same type.
> >
> > Also, there seems to be a substantial drop-off in the performance
> > of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> > 40% on 4CPU)
> >
>
> Well one would expect writes to be much faster than reads because writes
> usually do not involve performing physical I/O, and when pagecache
> writeback finally happens it has vastly more data to work with and hence
> can schedule I/O more efficiently.
>
> Unless you are using O_SYNC or fsync(), in which case ignore the above.
It should be doing an fsync() between the write and read stages. For all
other types of reads the performance is substantially better than the
writes.
2.6.9-rc2-mm4
Direct Random write: 11.17 MB/s
Direct Random read: 44.71 MB/s
>
> The regression within random write performance is unexpected. Can you
> please provide a URL to the current version of the test tool, and a
> description of how you are invoking it? What sort of I/O system, what
> filesystem, etc.
I am running aio-stress from Chris Mason; you can get it from OSDL's
aio-stress test kit:
bk clone bk://developer.osdl.org/stp-tests/aio-stress
This is how I call it (with only one file):
aio-stress -o 2 -o 3 -s 4g -r 64k -l -L <files_name>
The 2 CPU host has MegaRAID, with 5 18G disks, striped. The 4CPU has
Adaptec 2200S x2, with 5 18G disks, striped.
The filesystem is ext2.
>
> Thanks.
>
Judith Lebzelter wrote:
> Hello;
>
> I am running aio-stress on the most recent kernels and have
> found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
> performance of buffered random reads is poor compared to the
> buffered random writes:
>
> 2.6.8 2.6.9-rc2 2.6.9-rc2-mm4
> --------------------------------------------
> random write 35.66 MB/s 34.80 MB/s 29.89 MB/s
> random read 7.69 MB/s 7.50 MB/s 7.68 MB/s
>
> ** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
>
>
> This shows up on our 4CPU host as well. (striped AACRAID.4G
> RAM. 8G File):
> 2.6.9-rc2 2.6.9-rc2-mm4 2.6.9-rc2-mm1
> -------------------------------------------
> random write 31.36 MB/s 18.92 MB/s 18.97 MB/s
> random read 11.13 MB/s 9.74 MB/s 11.05 MB/s
>
>
> There seems to be an issue with the reads. Usually, reads
> should be at least as fast as writes of the same type.
Usually not, since the write goes to cache in most cases while the read
must really happen. The 40% slowdown is troubling, though, that's
unexpected.
--
-bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me
On Wed, 29 Sep 2004, William Lee Irwin III wrote:
> On Wed, Sep 29, 2004 at 04:29:08PM -0700, Judith Lebzelter wrote:
> > There seems to be an issue with the reads. Usually, reads
> > should be at least as fast as writes of the same type.
> > Also, there seems to be a substantial drop-off in the performance
> > of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> > 40% on 4CPU)
>
> Okay, is it cpu time or idle/iowait? If it's cpu time, where do
> profiles show it appears?
Th CPU is not that busy:
2.6.9-rc2-mm4 Results and iostat outputs:
http://khack.osdl.org/stp/297714/
http://khack.osdl.org/stp/297714/results/bufferrand/iostat.txt
2.6.9-rc2 Results and iostat outputs:
http://khack.osdl.org/stp/297545/
http://khack.osdl.org/stp/297545/results/bufferrand/iostat.txt
The iostat has the write stats followed by the reads, taken every 15
seconds.
>
>
> -- wli
>