Hello,
During our application testing, we noticed that our application (that
operates directly on a LVM volume) we noticed that it seems the read
data does not go into any cache.
Now we did some tests using dd blocksize=1M count=1000:
Using dd directly on the /dev/daten/testing lvm volume, we read about 95
MBytes/Seconds. Issuing multiple dds in sequence gives little variance in IO
speed (between 90 and 100 MB/sec).
When we create a file system on this volume, and mount it, and we create
a 1G file there, the dd gives us the same 95 MB/sec on the first read
after the mount, and approx. 480 MB/Sec on subsequent reads.
The machine runs Kernel 2.6.5 SMP on a dual SMT Itanium HP Box.
This lead us to the conclusion that block devices do not cache, but the
filesystem does. But subsequently, I ran some tests on my developer
machine (Pentium 4 Mobile Laptop).
dd'ing 16MB from an usb 1.1 stick present as /dev/sda, I got about
900k/sec on every read, so this seems to be uncached access, too.
When dd'ing 100MB from /dev/hda5, the first read gives about
22MBytes/Sek (which seems okay for a 2.5" IDE Disk), but subsequend
reads give about 389MBytes/sek (which is impossible to achieve using
such hardware). Interestingly, this happens on a mounted partition,
while when unmounting the partition, caching does not work. But for the
/dev/daten/testing above, mounting a filesystem on the partition does
not help in caching.
Can anyone give us a hint what's happening here, or even tell us how to
use a block device via the kernel caching (and memory mapping does not
work easily, as the production lvm volume is about 600 Gig on said
32-bit X86 machine.
Thanks,
Markus
PS: I subscribed to the list, so no Cc:-ing required.
--
markus schaber | dipl. informatiker
logi-track ag | rennweg 14-16 | ch 8001 z?rich
phone +41-43-888 62 52 | fax +41-43-888 62 53
mailto:[email protected] | http://www.logi-track.com
On Jun 30, 2004 00:20 +0200, Markus Schaber wrote:
> During our application testing, we noticed that our application (that
> operates directly on a LVM volume) we noticed that it seems the read
> data does not go into any cache.
>
> Now we did some tests using dd blocksize=1M count=1000:
>
> Using dd directly on the /dev/daten/testing lvm volume, we read about 95
> MBytes/Seconds. Issuing multiple dds in sequence gives little variance in IO
> speed (between 90 and 100 MB/sec).
>
> When we create a file system on this volume, and mount it, and we create
> a 1G file there, the dd gives us the same 95 MB/sec on the first read
> after the mount, and approx. 480 MB/Sec on subsequent reads.
>
> This lead us to the conclusion that block devices do not cache, but the
> filesystem does. But subsequently, I ran some tests on my developer
> machine (Pentium 4 Mobile Laptop).
>
> When dd'ing 100MB from /dev/hda5, the first read gives about
> 22MBytes/Sek (which seems okay for a 2.5" IDE Disk), but subsequend
> reads give about 389MBytes/sek (which is impossible to achieve using
> such hardware). Interestingly, this happens on a mounted partition,
> while when unmounting the partition, caching does not work. But for the
> /dev/daten/testing above, mounting a filesystem on the partition does
> not help in caching.
When you close the block device it flushes the cache for that device (inode).
If you kept the device open in some way (e.g. "sleep 10000000 < /dev/hda5")
then it should start caching the data between dd runs.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/
Markus Schaber wrote:
> This lead us to the conclusion that block devices do not cache, but the
> filesystem does. But subsequently, I ran some tests on my developer
> machine (Pentium 4 Mobile Laptop).
I had kernel experts repeatedly insist to me that block devices were
cached, while all of my tests (using dd to or from, say, /dev/sda1 or
whatever) indicated that there was absolutely no caching whatsoever.
In my experience, reads and writes to block devices are uncached, while
filesystem stuff IS cached.
Andreas Dilger wrote:
>
>
> When you close the block device it flushes the cache for that device (inode).
> If you kept the device open in some way (e.g. "sleep 10000000 < /dev/hda5")
> then it should start caching the data between dd runs.
>
Ok, now THIS makes sense.
Hi,
On Tue, 29 Jun 2004 16:46:22 -0600
Andreas Dilger <[email protected]> wrote:
>>[block device caching problems]
> When you close the block device it flushes the cache for that device
> (inode). If you kept the device open in some way (e.g. "sleep 10000000
> < /dev/hda5") then it should start caching the data between dd runs.
This sounds reasonable, and it works using hda5 on my developer machine:
root@kingfisher:~# dd if=/dev/hda5 of=/dev/null bs=1M count=100
100+0 Datens?tze ein
100+0 Datens?tze aus
104857600 bytes transferred in 4,644529 seconds (22576584 bytes/sec)
root@kingfisher:~# dd if=/dev/hda5 of=/dev/null bs=1M count=100
100+0 Datens?tze ein
100+0 Datens?tze aus
104857600 bytes transferred in 4,688006 seconds (22367207 bytes/sec)
root@kingfisher:~# sleep 1000000 </dev/hda5 &
[1] 17321
root@kingfisher:~# dd if=/dev/hda5 of=/dev/null bs=1M count=100
100+0 Datens?tze ein
100+0 Datens?tze aus
104857600 bytes transferred in 4,662113 seconds (22491433 bytes/sec)
root@kingfisher:~# dd if=/dev/hda5 of=/dev/null bs=1M count=100
100+0 Datens?tze ein
100+0 Datens?tze aus
104857600 bytes transferred in 0,271807 seconds (385779610 bytes/sec)
And it works on my USB memory stick:
root@kingfisher:~# dd if=/dev/sda of=/dev/null bs=1M count=100
31+1 Datens?tze ein
31+1 Datens?tze aus
32768000 bytes transferred in 36,011661 seconds (909927 bytes/sec)
root@kingfisher:~# dd if=/dev/sda of=/dev/null bs=1M count=100
31+1 Datens?tze ein
31+1 Datens?tze aus
32768000 bytes transferred in 37,133379 seconds (882441 bytes/sec)
root@kingfisher:~# sleep 1000000 </dev/sda &
[1] 17375
root@kingfisher:~# dd if=/dev/sda of=/dev/null bs=1M count=100
31+1 Datens?tze ein
31+1 Datens?tze aus
32768000 bytes transferred in 36,004170 seconds (910117 bytes/sec)
root@kingfisher:~# dd if=/dev/sda of=/dev/null bs=1M count=100
31+1 Datens?tze ein
31+1 Datens?tze aus
32768000 bytes transferred in 0,088144 seconds (371755027 bytes/sec)
But it seems to fail on our production machine LVM volume:
bear:~# dd if=/dev/daten/testing of=/dev/null bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 10.779081 seconds (97278793 bytes/sec)
bear:~# dd if=/dev/daten/testing of=/dev/null bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 10.773274 seconds (97331229 bytes/sec)
bear:~# sleep 1000000 </dev/daten/testing &
[1] 23588
bear:~# dd if=/dev/daten/testing of=/dev/null bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 11.030774 seconds (95059149 bytes/sec)
bear:~# dd if=/dev/daten/testing of=/dev/null bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 11.027699 seconds (95085656 bytes/sec)
top on bear displays:
top - 08:23:31 up 68 days, 5:14, 8 users, load average: 0.08, 0.08, 0.03
Tasks: 81 total, 1 running, 79 sleeping, 0 stopped, 1 zombie
Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.4% id, 0.0% wa, 0.6% hi, 0.0% si
Cpu1 : 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 3953624k total, 1120508k used, 2833116k free, 800248k buffers
Swap: 1048568k total, 0k used, 1048568k free, 24932k cached
So there clearly is enough free RAM to buffer 1 Gig of data (which, as I
said, works for a file when mounting /dev/daten/testing with ext3).
Did we do something wrong in our LVM setup? Do you need more info (such
as the output of some lv tools or the kernel config)?
Thanks for your efforts,
Markus Schaber
--
markus schaber | dipl. informatiker
logi-track ag | rennweg 14-16 | ch 8001 z?rich
phone +41-43-888 62 52 | fax +41-43-888 62 53
mailto:[email protected] | http://www.logi-track.com
Timothy Miller wrote:
>
>
> Markus Schaber wrote:
>
>> This lead us to the conclusion that block devices do not cache, but the
>> filesystem does. But subsequently, I ran some tests on my developer
>> machine (Pentium 4 Mobile Laptop).
>
>
>
> I had kernel experts repeatedly insist to me that block devices were
> cached, while all of my tests (using dd to or from, say, /dev/sda1 or
> whatever) indicated that there was absolutely no caching whatsoever.
Well, any cache is dropped when the device is closed. "dd" closes the device
when it finishes.
Try a program that reads the same two blocks (spaced videly apart)
over and over from the same open file descriptor. With _no_ caching
you'll see
the drive seeking all the time. With caching, you won't.
Helge Hafting