LinuxLists.cc - Abysmal disk performance, how to debug?

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

Hi,

On Sat, Jan 20, 2007 at 07:20:53PM +0200, Ismail D?nmez wrote:
> Hi all,
>
> I own a Sony Vaio VGN-FS285B and disk performance to say at least is very very
> slow. Writing 1 GB data makes the laptop unresponsive. Here is some data
> identifying the drive. Hope someone can tell me how to debug and find out
> whats the problem.
>
> FWIW since 2.6.16 the problem is same and I didn't test with 2.4 kernels. Here
> is some data. And I tested with xfs & ext3. Both slow slow disk writes.
>
> vaio cartman # hdparm /dev/hda
>
> /dev/hda:
> multcount = 16 (on)
> IO_support = 1 (32-bit)
> unmaskirq = 1 (on)
> using_dma = 1 (on)
> keepsettings = 0 (off)
> readonly = 0 (off)
> readahead = 256 (on)
> geometry = 65535/16/63, sectors = 156301488, start = 0
> vaio cartman # hdparm -i /dev/hda
>
> /dev/hda:
>
> Model=FUJITSU MHV2080AT, FwRev=00000096, SerialNo=NS56T58270LE
> Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
> RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
> BuffType=DualPortCache, BuffSize=8192kB, MaxMultSect=16, MultSect=16
> CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=156301488
> IORDY=yes, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
> PIO modes: pio0 pio1 pio2 pio3 pio4
> DMA modes: mdma0 mdma1 mdma2
> UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma3 udma4 *udma5
> AdvancedPM=yes: mode=0x80 (128) WriteCache=enabled
> Drive conforms to: ATA/ATAPI-6 T13 1410D revision 3a: ATA/ATAPI-2
> ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6
>
> * signifies the current active mode
>
>
> vaio cartman # hdparm -tT /dev/hda
>
> /dev/hda:
> Timing cached reads: 1576 MB in 2.00 seconds = 788.18 MB/sec
> Timing buffered disk reads: 74 MB in 3.01 seconds = 24.55 MB/sec
>
>
> [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1,1 GB) copied, 77,2809 s, 13,9 MB/s
>
> real 1m17.482s
> user 0m0.003s
> sys 0m2.350s

That's not bad at all ! I suspect that if your system becomes unresponsive,
it's because real writes start when the cache is full. And if you fill
512 MB of RAM with data that you then need to flush on disk at 14 MB/s, it
can take about 40 seconds during which it might be difficult to do anything.

Try lowering the cache flush starting point to about 10 MB if you want
(2% of 512 MB) :

# echo 2 >/proc/sys/vm/dirty_ratio
# echo 2 >/proc/sys/vm/dirty_background_ratio

I see nothing wrong in your measures nor messages.

Regards,
Willy

2007-01-20 17:53:05

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

20 Oca 2007 Cts 19:45 tarihinde şunları yazmıştınız:
[...]
> > vaio cartman # hdparm -tT /dev/hda
> >
> > /dev/hda:
> > Timing cached reads: 1576 MB in 2.00 seconds = 788.18 MB/sec
> > Timing buffered disk reads: 74 MB in 3.01 seconds = 24.55 MB/sec
> >
> >
> > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > 1024+0 records in
> > 1024+0 records out
> > 1073741824 bytes (1,1 GB) copied, 77,2809 s, 13,9 MB/s
> >
> > real 1m17.482s
> > user 0m0.003s
> > sys 0m2.350s
>
> That's not bad at all ! I suspect that if your system becomes unresponsive,
> it's because real writes start when the cache is full. And if you fill
> 512 MB of RAM with data that you then need to flush on disk at 14 MB/s, it
> can take about 40 seconds during which it might be difficult to do
> anything.
>
> Try lowering the cache flush starting point to about 10 MB if you want
> (2% of 512 MB) :
>
> # echo 2 >/proc/sys/vm/dirty_ratio
> # echo 2 >/proc/sys/vm/dirty_background_ratio

After that I get,

[~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1,1 GB) copied, 41,7005 s, 25,7 MB/s

real 0m41.926s
user 0m0.007s
sys 0m2.500s

not bad! thanks :)

Regards,
ismail

2007-01-20 18:03:50

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, Jan 20, 2007 at 07:52:53PM +0200, Ismail D?nmez wrote:
> 20 Oca 2007 Cts 19:45 tarihinde ??unlar?? yazm????t??n??z:
> [...]
> > > vaio cartman # hdparm -tT /dev/hda
> > >
> > > /dev/hda:
> > > Timing cached reads: 1576 MB in 2.00 seconds = 788.18 MB/sec
> > > Timing buffered disk reads: 74 MB in 3.01 seconds = 24.55 MB/sec
> > >
> > >
> > > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > > 1024+0 records in
> > > 1024+0 records out
> > > 1073741824 bytes (1,1 GB) copied, 77,2809 s, 13,9 MB/s
> > >
> > > real 1m17.482s
> > > user 0m0.003s
> > > sys 0m2.350s
> >
> > That's not bad at all ! I suspect that if your system becomes unresponsive,
> > it's because real writes start when the cache is full. And if you fill
> > 512 MB of RAM with data that you then need to flush on disk at 14 MB/s, it
> > can take about 40 seconds during which it might be difficult to do
> > anything.
> >
> > Try lowering the cache flush starting point to about 10 MB if you want
> > (2% of 512 MB) :
> >
> > # echo 2 >/proc/sys/vm/dirty_ratio
> > # echo 2 >/proc/sys/vm/dirty_background_ratio
>
> After that I get,
>
> [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1,1 GB) copied, 41,7005 s, 25,7 MB/s
>
> real 0m41.926s
> user 0m0.007s
> sys 0m2.500s
>
>
> not bad! thanks :)

It is not expected to increase write performance, but it should help
you do something else during that time, or also give more responsiveness
to Ctrl-C. It is possible that you have fast and slow RAM, or that your
video card uses shared memory which slows down some parts of memory
which are not used anymore with those parameters.

Regards,
Willy

2007-01-20 18:06:52

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

20 Oca 2007 Cts 20:03 tarihinde şunları yazmıştınız:
> On Sat, Jan 20, 2007 at 07:52:53PM +0200, Ismail Dönmez wrote:
> > 20 Oca 2007 Cts 19:45 tarihinde ??unlar?? yazm????t??n??z:
> > [...]
> >
> > > > vaio cartman # hdparm -tT /dev/hda
> > > >
> > > > /dev/hda:
> > > > Timing cached reads: 1576 MB in 2.00 seconds = 788.18 MB/sec
> > > > Timing buffered disk reads: 74 MB in 3.01 seconds = 24.55 MB/sec
> > > >
> > > >
> > > > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > > > 1024+0 records in
> > > > 1024+0 records out
> > > > 1073741824 bytes (1,1 GB) copied, 77,2809 s, 13,9 MB/s
> > > >
> > > > real 1m17.482s
> > > > user 0m0.003s
> > > > sys 0m2.350s
> > >
> > > That's not bad at all ! I suspect that if your system becomes
> > > unresponsive, it's because real writes start when the cache is full.
> > > And if you fill 512 MB of RAM with data that you then need to flush on
> > > disk at 14 MB/s, it can take about 40 seconds during which it might be
> > > difficult to do anything.
> > >
> > > Try lowering the cache flush starting point to about 10 MB if you want
> > > (2% of 512 MB) :
> > >
> > > # echo 2 >/proc/sys/vm/dirty_ratio
> > > # echo 2 >/proc/sys/vm/dirty_background_ratio
> >
> > After that I get,
> >
> > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > 1024+0 records in
> > 1024+0 records out
> > 1073741824 bytes (1,1 GB) copied, 41,7005 s, 25,7 MB/s
> >
> > real 0m41.926s
> > user 0m0.007s
> > sys 0m2.500s
> >
> >
> > not bad! thanks :)
>
> It is not expected to increase write performance, but it should help
> you do something else during that time, or also give more responsiveness
> to Ctrl-C. It is possible that you have fast and slow RAM, or that your
> video card uses shared memory which slows down some parts of memory
> which are not used anymore with those parameters.

Thanks I will try to upgrade RAM but for now at least responsiveness seems to
be better.

Regards,
ismail

2007-01-20 19:44:45

by Sunil Naidu

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On 1/20/07, Willy Tarreau <[email protected]> wrote:
> > > >
>
> It is not expected to increase write performance, but it should help
> you do something else during that time, or also give more responsiveness
> to Ctrl-C. It is possible that you have fast and slow RAM, or that your
> video card uses shared memory which slows down some parts of memory
> which are not used anymore with those parameters.

I did test some SATA drives, am getting these value for 2.6.20-rc5:-

[sukhoi@Typhoon ~]$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 21.0962 seconds, 50.9 MB/s

What can you suggest here w.r.t my RAM & disk?

> Willy

Thanks,

~Akula2

2007-01-20 19:56:24

by Stephen Clark

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

Sunil Naidu wrote:

>On 1/20/07, Willy Tarreau <[email protected]> wrote:
>
>
>>It is not expected to increase write performance, but it should help
>>you do something else during that time, or also give more responsiveness
>>to Ctrl-C. It is possible that you have fast and slow RAM, or that your
>>video card uses shared memory which slows down some parts of memory
>>which are not used anymore with those parameters.
>>
>>
>
>I did test some SATA drives, am getting these value for 2.6.20-rc5:-
>
>[sukhoi@Typhoon ~]$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
>1024+0 records in
>1024+0 records out
>1073741824 bytes (1.1 GB) copied, 21.0962 seconds, 50.9 MB/s
>
>What can you suggest here w.r.t my RAM & disk?
>
>
>
>>Willy
>>
>>
>
>Thanks,
>
>~Akula2
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to [email protected]
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
>
>
>
Hi,
whitebook vbi s96f core 2 duo t5600 2gb hitachi ATA HTS721060G9AT00
using libata
time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.0092 seconds, 107 MB/s

real 0m10.196s
user 0m0.004s
sys 0m3.440s

--

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety." (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases." (Thomas Jefferson)

2007-01-20 20:05:45

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sun, Jan 21, 2007 at 01:14:41AM +0530, Sunil Naidu wrote:
> On 1/20/07, Willy Tarreau <[email protected]> wrote:
> >> > >
> >
> >It is not expected to increase write performance, but it should help
> >you do something else during that time, or also give more responsiveness
> >to Ctrl-C. It is possible that you have fast and slow RAM, or that your
> >video card uses shared memory which slows down some parts of memory
> >which are not used anymore with those parameters.
>
> I did test some SATA drives, am getting these value for 2.6.20-rc5:-
>
> [sukhoi@Typhoon ~]$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 21.0962 seconds, 50.9 MB/s
>
> What can you suggest here w.r.t my RAM & disk?

I don't suggest anything, this is already very good. The only goal of
reducing memory write cache is to get a more responsive system when
dumping massive amounts of data to disks, like above, because when
the system starts flushing the caches, you can only wait for it to
finish. But those tests are not realistic loads. A desktop and most
servers will benefit from large caches. But *some* workloads will
benefit from smaller caches if they consist in writing continuous
streams (eg: tcpdump or video recorders). What I suggested to the
user above was a way to get the system more responsive during his
test. It should not have changed the throughput at all if the
hardware was not a bit strange (well, it's a VAIO after all, I've
had one too, fortunately it died one month after the warranty,
putting an end to all my problems...).

Regards,
Willy

2007-01-20 20:09:49

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, Jan 20, 2007 at 02:56:20PM -0500, Stephen Clark wrote:
> Sunil Naidu wrote:
>
> >On 1/20/07, Willy Tarreau <[email protected]> wrote:
> >
> >
> >>It is not expected to increase write performance, but it should help
> >>you do something else during that time, or also give more responsiveness
> >>to Ctrl-C. It is possible that you have fast and slow RAM, or that your
> >>video card uses shared memory which slows down some parts of memory
> >>which are not used anymore with those parameters.
> >>
> >>
> >
> >I did test some SATA drives, am getting these value for 2.6.20-rc5:-
> >
> >[sukhoi@Typhoon ~]$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> >1024+0 records in
> >1024+0 records out
> >1073741824 bytes (1.1 GB) copied, 21.0962 seconds, 50.9 MB/s
> >
> >What can you suggest here w.r.t my RAM & disk?
> >
> >
> >
> >>Willy
> >>
> >>
> >
> >Thanks,
> >
> >~Akula2
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >the body of a message to [email protected]
> >More majordomo info at http://vger.kernel.org/majordomo-info.html
> >Please read the FAQ at http://www.tux.org/lkml/
> >
> >
> >
> Hi,
> whitebook vbi s96f core 2 duo t5600 2gb hitachi ATA HTS721060G9AT00
> using libata
> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 10.0092 seconds, 107 MB/s
>
> real 0m10.196s
> user 0m0.004s
> sys 0m3.440s

You have too much RAM, it's possible that writes did not complete before
the end of your measurement. Try this instead :

$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024 | sync

Willy

2007-01-20 20:10:26

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, 20 Jan 2007, Ismail Dönmez wrote:

> 20 Oca 2007 Cts 19:45 tarihinde şunları yazmıştınız:
> [...]
> > > vaio cartman # hdparm -tT /dev/hda
> > >
> > > /dev/hda:
> > > Timing cached reads: 1576 MB in 2.00 seconds = 788.18 MB/sec
> > > Timing buffered disk reads: 74 MB in 3.01 seconds = 24.55 MB/sec
> > >
> > >
> > > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > > 1024+0 records in
> > > 1024+0 records out
> > > 1073741824 bytes (1,1 GB) copied, 77,2809 s, 13,9 MB/s
> > >
> > > real 1m17.482s
> > > user 0m0.003s
> > > sys 0m2.350s
> >
> > That's not bad at all ! I suspect that if your system becomes unresponsive,
> > it's because real writes start when the cache is full. And if you fill
> > 512 MB of RAM with data that you then need to flush on disk at 14 MB/s, it
> > can take about 40 seconds during which it might be difficult to do
> > anything.
> >
> > Try lowering the cache flush starting point to about 10 MB if you want
> > (2% of 512 MB) :
> >
> > # echo 2 >/proc/sys/vm/dirty_ratio
> > # echo 2 >/proc/sys/vm/dirty_background_ratio
>
> After that I get,
>
> [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1,1 GB) copied, 41,7005 s, 25,7 MB/s
>
> real 0m41.926s
> user 0m0.007s
> sys 0m2.500s
>
>
> not bad! thanks :)

Note that these dd "benchmarks" are completely bogus, because the data
doesn't actually get written to disk in that time. For some enlightening
data, try

time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024; time sync

The dd returns as soon as all data could be buffered in RAM. Only sync
will show how long it takes to actually write out the data to disk.
also explains why you see better results is writeout starts earlier.

Tim

2007-01-20 20:11:57

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

20 Oca 2007 Cts 22:05 tarihinde, Willy Tarreau şunları yazmıştı:
> On Sun, Jan 21, 2007 at 01:14:41AM +0530, Sunil Naidu wrote:
> > On 1/20/07, Willy Tarreau <[email protected]> wrote:
[...]
> It should not have changed the throughput at all if the
> hardware was not a bit strange (well, it's a VAIO after all, I've
> had one too, fortunately it died one month after the warranty,
> putting an end to all my problems...).

How true is this, VAIO sucks on Linux :'(

Regards,
ismail

2007-01-20 20:16:24

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

20 Oca 2007 Cts 22:10 tarihinde, Tim Schmielau şunları yazmıştı:
[...]
>
> Note that these dd "benchmarks" are completely bogus, because the data=20
> doesn't actually get written to disk in that time. For some enlightening=20
> data, try
>
> time dd if=3D/dev/zero of=3D/tmp/1GB bs=3D1M count=3D1024; time sync
>
> The dd returns as soon as all data could be buffered in RAM. Only sync=20
> will show how long it takes to actually write out the data to disk.
> also explains why you see better results is writeout starts earlier.

Still not that bad:

[~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024;sync
1024+0 records in
1024+0 records out
1073741824 bytes (1,1 GB) copied, 53,3194 s, 20,1 MB/s

real 0m53.517s
user 0m0.003s
sys 0m3.193s

2007-01-20 20:19:51

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, Jan 20, 2007 at 10:16:15PM +0200, Ismail D?nmez wrote:
> 20 Oca 2007 Cts 22:10 tarihinde, Tim Schmielau ??unlar?? yazm????t??:
> [...]
> >
> > Note that these dd "benchmarks" are completely bogus, because the data=20
> > doesn't actually get written to disk in that time. For some enlightening=20
> > data, try
> >
> > time dd if=3D/dev/zero of=3D/tmp/1GB bs=3D1M count=3D1024; time sync
> >
> > The dd returns as soon as all data could be buffered in RAM. Only sync=20
> > will show how long it takes to actually write out the data to disk.
> > also explains why you see better results is writeout starts earlier.
>
> Still not that bad:
>
> [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024;sync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1,1 GB) copied, 53,3194 s, 20,1 MB/s
>
> real 0m53.517s
> user 0m0.003s
> sys 0m3.193s

No, your measure is wrong because time measures "dd" and sync is done
after. Either use Tim's method (time sync) or the one I proposed in
previous mail (time dd | sync). Anyway, in your situation with a very
small buffer, this should not change by more than half a second or so.

Willy

2007-01-20 20:21:34

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, 20 Jan 2007, Ismail Dönmez wrote:

> 20 Oca 2007 Cts 22:10 tarihinde, Tim Schmielau şunları yazmıştı:
> >
> > Note that these dd "benchmarks" are completely bogus, because the data=20
> > doesn't actually get written to disk in that time. For some enlightening=20
> > data, try
> >
> > time dd if=3D/dev/zero of=3D/tmp/1GB bs=3D1M count=3D1024; time sync
^^^^
> >
> > The dd returns as soon as all data could be buffered in RAM. Only sync=20
> > will show how long it takes to actually write out the data to disk.
> > also explains why you see better results is writeout starts earlier.
>
> Still not that bad:
>
> [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024;sync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1,1 GB) copied, 53,3194 s, 20,1 MB/s
>
> real 0m53.517s
> user 0m0.003s
> sys 0m3.193s
>

That's not the point, you still measured the same as before (but you might
have noticed that, after printing the results, the shell prompt took some
time to appear). I appended "time sync" to the command to show that
(depending on the amount of available memory) actually most of the time
is spent in the "sync", not the "dd".

Tim

2007-01-20 20:28:50

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

Hi Tim,

On Sat, Jan 20, 2007 at 09:10:22PM +0100, Tim Schmielau wrote:
> On Sat, 20 Jan 2007, Ismail D?nmez wrote:
>
> > 20 Oca 2007 Cts 19:45 tarihinde ??unlar?? yazm????t??n??z:
> > [...]
> > > > vaio cartman # hdparm -tT /dev/hda
> > > >
> > > > /dev/hda:
> > > > Timing cached reads: 1576 MB in 2.00 seconds = 788.18 MB/sec
> > > > Timing buffered disk reads: 74 MB in 3.01 seconds = 24.55 MB/sec
> > > >
> > > >
> > > > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > > > 1024+0 records in
> > > > 1024+0 records out
> > > > 1073741824 bytes (1,1 GB) copied, 77,2809 s, 13,9 MB/s
> > > >
> > > > real 1m17.482s
> > > > user 0m0.003s
> > > > sys 0m2.350s
> > >
> > > That's not bad at all ! I suspect that if your system becomes unresponsive,
> > > it's because real writes start when the cache is full. And if you fill
> > > 512 MB of RAM with data that you then need to flush on disk at 14 MB/s, it
> > > can take about 40 seconds during which it might be difficult to do
> > > anything.
> > >
> > > Try lowering the cache flush starting point to about 10 MB if you want
> > > (2% of 512 MB) :
> > >
> > > # echo 2 >/proc/sys/vm/dirty_ratio
> > > # echo 2 >/proc/sys/vm/dirty_background_ratio
> >
> > After that I get,
> >
> > [~]> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024
> > 1024+0 records in
> > 1024+0 records out
> > 1073741824 bytes (1,1 GB) copied, 41,7005 s, 25,7 MB/s
> >
> > real 0m41.926s
> > user 0m0.007s
> > sys 0m2.500s
> >
> >
> > not bad! thanks :)
>
> Note that these dd "benchmarks" are completely bogus, because the data
> doesn't actually get written to disk in that time. For some enlightening
> data, try
>
> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024; time sync
>
> The dd returns as soon as all data could be buffered in RAM. Only sync
> will show how long it takes to actually write out the data to disk.

While I 100% agree with you on this, I'd like to note that I don't agree
with the following statement :

> also explains why you see better results is writeout starts earlier.

The results should be better when writeout starts later since most of
the transfer will have been performed at RAM speed. That's what happens
with the user above with 2 GB RAM. But in case of the VAIO with 512 MB,
there's really something strange IMHO. I suspect it has two RAM areas,
one fast and one slow (probably one two large non-cacheable area for a
shared video or such a crap, which can be avoided when reducing the
cache size).

Best regards,
Willy

2007-01-20 20:28:59

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, 20 Jan 2007, Willy Tarreau wrote:

> Anyway, in your situation with a very small buffer, this should not
> change by more than half a second or so.

Well, his buffer is not small. He has half a GB of RAM, so when
writing 1 GB the buffer would roughly double the dd speed, exactly as he
has shown us.

Anyways, instead of always just posting similar answers to yours, I'll
have dinner now. :-)

Tim

2007-01-20 20:32:11

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, Jan 20, 2007 at 09:28:57PM +0100, Tim Schmielau wrote:
> On Sat, 20 Jan 2007, Willy Tarreau wrote:
>
> > Anyway, in your situation with a very small buffer, this should not
> > change by more than half a second or so.
>
> Well, his buffer is not small. He has half a GB of RAM, so when
> writing 1 GB the buffer would roughly double the dd speed, exactly as he
> has shown us.

yes, but he sees the opposite : when using half of the memory for the
buffer, he has low speed. When shorting cache size to 2% only, he doubles
his speed.

> Anyways, instead of always just posting similar answers to yours, I'll
> have dinner now. :-)

:-)
Willy

2007-01-20 20:39:27

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, 20 Jan 2007, Willy Tarreau wrote:
> On Sat, Jan 20, 2007 at 09:10:22PM +0100, Tim Schmielau wrote:
> >
> > Note that these dd "benchmarks" are completely bogus, because the data
> > doesn't actually get written to disk in that time. For some enlightening
> > data, try
> >
> > time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024; time sync
> >
> > The dd returns as soon as all data could be buffered in RAM. Only sync
> > will show how long it takes to actually write out the data to disk.
>
> While I 100% agree with you on this, I'd like to note that I don't agree
> with the following statement :
>
> > also explains why you see better results is writeout starts earlier.
>
> The results should be better when writeout starts later since most of
> the transfer will have been performed at RAM speed. That's what happens
> with the user above with 2 GB RAM. But in case of the VAIO with 512 MB,
> there's really something strange IMHO. I suspect it has two RAM areas,
> one fast and one slow (probably one two large non-cacheable area for a
> shared video or such a crap, which can be avoided when reducing the
> cache size).

No - the earlier the writeout starts, the earlier he will have enough free
RAM to finish the dd command by buffering the remaining data.

Note that we did not cap the amount of buffers, just started to write out
earlier.

Tim

2007-01-20 21:12:44

by Sunil Naidu

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On 1/21/07, Tim Schmielau <[email protected]> wrote:
>
> Note that these dd "benchmarks" are completely bogus, because the data
> doesn't actually get written to disk in that time. For some enlightening
> data, try
>
> time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024; time sync
>
> The dd returns as soon as all data could be buffered in RAM. Only sync
> will show how long it takes to actually write out the data to disk.
> also explains why you see better results is writeout starts earlier.

I am still getting better I feel:

[sukhoi@Typhoon ~]$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024; time sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 19.5007 seconds, 55.1 MB/s

real 0m20.439s
user 0m0.004s
sys 0m4.535s

real 0m4.625s
user 0m0.000s
sys 0m0.125s

[sukhoi@Typhoon ~]$ time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024 | sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 20.8707 seconds, 51.4 MB/s

real 0m22.449s
user 0m0.002s
sys 0m4.922s

Linux used here is not 2.6.20-rc5, but it's a FC6 2.6.19 binary. Shall
post the results with 2.6.20-rc5.

BTW, does the results vary with a customized kernel (configured w.r.t
Processor & Hardware) than a generic kernel like FC6?

Are there any other such test cases?

>
> Tim
>

Thanks,

~Akula2

2007-01-20 21:24:46

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, Jan 20, 2007 at 09:39:25PM +0100, Tim Schmielau wrote:
> On Sat, 20 Jan 2007, Willy Tarreau wrote:
> > On Sat, Jan 20, 2007 at 09:10:22PM +0100, Tim Schmielau wrote:
> > >
> > > Note that these dd "benchmarks" are completely bogus, because the data
> > > doesn't actually get written to disk in that time. For some enlightening
> > > data, try
> > >
> > > time dd if=/dev/zero of=/tmp/1GB bs=1M count=1024; time sync
> > >
> > > The dd returns as soon as all data could be buffered in RAM. Only sync
> > > will show how long it takes to actually write out the data to disk.
> >
> > While I 100% agree with you on this, I'd like to note that I don't agree
> > with the following statement :
> >
> > > also explains why you see better results is writeout starts earlier.
> >
> > The results should be better when writeout starts later since most of
> > the transfer will have been performed at RAM speed. That's what happens
> > with the user above with 2 GB RAM. But in case of the VAIO with 512 MB,
> > there's really something strange IMHO. I suspect it has two RAM areas,
> > one fast and one slow (probably one two large non-cacheable area for a
> > shared video or such a crap, which can be avoided when reducing the
> > cache size).
>
> No - the earlier the writeout starts, the earlier he will have enough free
> RAM to finish the dd command by buffering the remaining data.

OK I see your point. While trying to show why I got you wrong, I in fact
demonstrated to myself that you were right :-)

For instance, let's say we have 500 MB cache at 1000 MB/s and a write out
threshold of 80% with a disk at 100 MB/s. Writing 1000 MB would produce
this pattern :

time data sent written dirty data
in sec from dd to disk in cache
0.0 0 MB 0 MB 0 MB
0.4 400 MB 0 MB 400 MB -> writeout starts
1.0 560 MB 60 MB 500 MB
5.4 1000 MB 500 MB 500 MB -> dd leaves.
10.4 1000 MB 1000 MB 0 MB -> write terminated.

-> avg dd speed = 1000/5.4 = 185 MB/s
avg disk speed = 1000/10.4 = 96 MB/s

Now with a lower writeout threshold of 2% (10 MB) :

time data sent written dirty data
in sec from dd to disk in cache
0.0 0 MB 0 MB 0 MB
0.01 10 MB 0 MB 10 MB -> writeout starts
1.0 599 MB 99 MB 500 MB
5.01 1000 MB 500 MB 500 MB -> dd leaves.
10.01 1000 MB 1000 MB 0 MB -> write terminated.

-> avg dd speed = 1000/5.01 = 199.6 MB/s
avg disk speed = 1000/10.01 = 99.9 MB/s

At least, numbers are not that much different to justify a one to two speed
ratio on the VAIO. The difference being caused by cache speed, it's clearly
possible that his RAM is definitely very very slow which would then explain
the difference.

----
> Note that we did not cap the amount of buffers, just started to write out
> earlier.
----

Indeed, that's what makes the whole difference. I was used to cap the amount
of buffers, but the behaviour here is different.

Thanks for your insight !
Willy

2007-01-20 21:41:34

[permalink] [raw]

Subject: Re: Abysmal disk performance, how to debug?

On Sat, 20 Jan 2007, Willy Tarreau wrote:
> On Sat, Jan 20, 2007 at 09:39:25PM +0100, Tim Schmielau wrote:
> > On Sat, 20 Jan 2007, Willy Tarreau wrote:
> > > On Sat, Jan 20, 2007 at 09:10:22PM +0100, Tim Schmielau wrote:
> > >
> > > > also explains why you see better results is writeout starts earlier.
> > >
> > > The results should be better when writeout starts later since most of
> > > the transfer will have been performed at RAM speed. That's what happens
> > > with the user above with 2 GB RAM. But in case of the VAIO with 512 MB,
> > > there's really something strange IMHO. I suspect it has two RAM areas,
> > > one fast and one slow (probably one two large non-cacheable area for a
> > > shared video or such a crap, which can be avoided when reducing the
> > > cache size).
> >
> > No - the earlier the writeout starts, the earlier he will have enough free
> > RAM to finish the dd command by buffering the remaining data.
>
> OK I see your point. While trying to show why I got you wrong, I in fact
> demonstrated to myself that you were right :-)
>
> For instance, let's say we have 500 MB cache at 1000 MB/s and a write out
> threshold of 80% with a disk at 100 MB/s. Writing 1000 MB would produce
> this pattern :
>
> time data sent written dirty data
> in sec from dd to disk in cache
> 0.0 0 MB 0 MB 0 MB
> 0.4 400 MB 0 MB 400 MB -> writeout starts
> 1.0 560 MB 60 MB 500 MB
> 5.4 1000 MB 500 MB 500 MB -> dd leaves.
> 10.4 1000 MB 1000 MB 0 MB -> write terminated.
>
> -> avg dd speed = 1000/5.4 = 185 MB/s
> avg disk speed = 1000/10.4 = 96 MB/s
>
>
> Now with a lower writeout threshold of 2% (10 MB) :
>
> time data sent written dirty data
> in sec from dd to disk in cache
> 0.0 0 MB 0 MB 0 MB
> 0.01 10 MB 0 MB 10 MB -> writeout starts
> 1.0 599 MB 99 MB 500 MB
> 5.01 1000 MB 500 MB 500 MB -> dd leaves.
> 10.01 1000 MB 1000 MB 0 MB -> write terminated.
>
> -> avg dd speed = 1000/5.01 = 199.6 MB/s
> avg disk speed = 1000/10.01 = 99.9 MB/s
>
> At least, numbers are not that much different to justify a one to two speed
> ratio on the VAIO. The difference being caused by cache speed, it's clearly
> possible that his RAM is definitely very very slow which would then explain
> the difference.
>
> ----
> > Note that we did not cap the amount of buffers, just started to write out
> > earlier.
> ----
>
> Indeed, that's what makes the whole difference. I was used to cap the amount
> of buffers, but the behaviour here is different.
>
> Thanks for your insight !

Thanks for being so humble in pointing out my logic is flawed. While the
Vaio certainly cannot write 1000GB/s to its RAM, it's disk is also quite
slow and the ratio of 10:1 for RAM:disk speed is presumably correct.
So we don't quite understand why dd in RAM is so slow for him.

Thanks,
Tim

2007-01-20 22:00:48