Hi all
After three days at Compaq's lab in Oslo, testing their medium-level
servers and storage systems with Linux, I've come to some sort of
conclusions, although these may be wrong. I also have come over a few
problems that I couln't find a good solution to.
* When running RAID from a Compaq Smart 5302/64 controller, software
RAID-5 is (slightly - ~15%) faster (on JBOD - each disk is configured as
a RAID-0 device with max - 256kB - stripe size) than the
hardware/controller based RAID-5. Both CPUs (1266MHz/512kB cache) are
maxed out by reading from software RAID-5 (???), giving me >= 107MB/s on
two SCSI-3 buses with six disks on each bus.
* Even though I can get up to 25 MB/s from each disk, I can't get more
than 107 MB/s on the whole bunch (12 drives). It doesn't help much to do
RAID-0 either. Don't understand anything ...
Thanks for all help.
roy
--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA
Computers are like air conditioners.
They stop working when you open Windows.
On Thu, Nov 15 2001, Roy Sigurd Karlsbakk wrote:
> Hi all
>
> After three days at Compaq's lab in Oslo, testing their medium-level
> servers and storage systems with Linux, I've come to some sort of
> conclusions, although these may be wrong. I also have come over a few
> problems that I couln't find a good solution to.
>
> * When running RAID from a Compaq Smart 5302/64 controller, software
> RAID-5 is (slightly - ~15%) faster (on JBOD - each disk is configured as
> a RAID-0 device with max - 256kB - stripe size) than the
> hardware/controller based RAID-5. Both CPUs (1266MHz/512kB cache) are
> maxed out by reading from software RAID-5 (???), giving me >= 107MB/s on
> two SCSI-3 buses with six disks on each bus.
>
> * Even though I can get up to 25 MB/s from each disk, I can't get more
> than 107 MB/s on the whole bunch (12 drives). It doesn't help much to do
> RAID-0 either. Don't understand anything ...
Could you please try and profile where the time is spent? Boot with
profile=2, and then do
# readprofile -r
# do I/O testing
# readprofile | sort -nr
--
Jens Axboe
> Could you please try and profile where the time is spent? Boot with
> profile=2, and then do
>
> # readprofile -r
> # do I/O testing
> # readprofile | sort -nr
I will.
However ... Is it normal for a server to max out 2xPIII 1266MHz CPUs by
reading from software RAID-5???
--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA
Computers are like air conditioners.
They stop working when you open Windows.
On Thu, Nov 15, 2001 at 05:32:29PM +0100, Roy Sigurd Karlsbakk wrote:
> > Could you please try and profile where the time is spent? Boot with
> > profile=2, and then do
> >
> > # readprofile -r
> > # do I/O testing
> > # readprofile | sort -nr
>
> I will.
>
> However ... Is it normal for a server to max out 2xPIII 1266MHz CPUs by
> reading from software RAID-5???
Certainly not.
Well, if you down-scale the experiment it's not. Reading 10.7 MB/sec
will not consume 10% of your two processors.
But queue systems are evil ;) I look forward to seeing the profile.
--
................................................................
: [email protected] : And I see the elder races, :
:.........................: putrid forms of man :
: Jakob ?stergaard : See him rise and claim the earth, :
: OZ9ABN : his downfall is at hand. :
:.........................:............{Konkhra}...............:
> Certainly not.
>
> Well, if you down-scale the experiment it's not. Reading 10.7 MB/sec
> will not consume 10% of your two processors.
>
> But queue systems are evil ;) I look forward to seeing the profile.
I'll post the pre- and post-profile here tomorrow...
roy
--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA
Computers are like air conditioners.
They stop working when you open Windows.
I guess the question I'd ask would be what kinda number do you expect from
the raid 5 stripe. 107MB/s sounds like a reasonable number from a two
channel u160 controller...
my previous best is around 89MB/s with 4 cheetah 15K 18gb drives raid/0
On Thu, 15 Nov 2001, Roy Sigurd Karlsbakk
wrote:
> Hi all
>
> After three days at Compaq's lab in Oslo, testing their medium-level
> servers and storage systems with Linux, I've come to some sort of
> conclusions, although these may be wrong. I also have come over a few
> problems that I couln't find a good solution to.
>
> * When running RAID from a Compaq Smart 5302/64 controller, software
> RAID-5 is (slightly - ~15%) faster (on JBOD - each disk is configured as
> a RAID-0 device with max - 256kB - stripe size) than the
> hardware/controller based RAID-5. Both CPUs (1266MHz/512kB cache) are
> maxed out by reading from software RAID-5 (???), giving me >= 107MB/s on
> two SCSI-3 buses with six disks on each bus.
>
> * Even though I can get up to 25 MB/s from each disk, I can't get more
> than 107 MB/s on the whole bunch (12 drives). It doesn't help much to do
> RAID-0 either. Don't understand anything ...
>
> Thanks for all help.
>
> roy
>
> --
> Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA
>
> Computers are like air conditioners.
> They stop working when you open Windows.
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
--------------------------------------------------------------------------
Joel Jaeggli [email protected]
Academic User Services [email protected]
PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E
--------------------------------------------------------------------------
It is clear that the arm of criticism cannot replace the criticism of
arms. Karl Marx -- Introduction to the critique of Hegel's Philosophy of
the right, 1843.
> I guess the question I'd ask would be what kinda number do you expect from
> the raid 5 stripe. 107MB/s sounds like a reasonable number from a two
> channel u160 controller...
I really don't know. My 'logical' mind told me I could get close-to
speed-per-disk * number-of-disks, but it might be wrong.
> my previous best is around 89MB/s with 4 cheetah 15K 18gb drives raid/0
I had 5 drives per SCSI bus and the controller was sitting alone on a
66MHz/64bit PCI hose. I really can't see where the bottleneck is!
roy
--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA
Computers are like air conditioners.
They stop working when you open Windows.
Roy Sigurd Karlsbakk wrote:
>>I guess the question I'd ask would be what kinda number do you expect from
>>the raid 5 stripe. 107MB/s sounds like a reasonable number from a two
>>channel u160 controller...
>>
>
>I really don't know. My 'logical' mind told me I could get close-to
>speed-per-disk * number-of-disks, but it might be wrong.
>
>
>>my previous best is around 89MB/s with 4 cheetah 15K 18gb drives raid/0
>>
>
>I had 5 drives per SCSI bus and the controller was sitting alone on a
>66MHz/64bit PCI hose. I really can't see where the bottleneck is!
>
# Hdparm
Do you use hdparm for your test ?
What gives hdparm -T (timing buffer-cache reading) ?
I believe it's kind of a hard ceiling...
<jealousy>
I use single channel PC66 sdram on a BX chipset here and have only 60
MB/s from buffer-cache on a SMP 2.4.13-ac2 kernel. Seeing a complain for
107 MB/s from a disk subsystem makes me feel somehow frustrated :-)
</jealousy>
# Hardware
Are all drives recognised as 160 MB/s capable devices ? SCSI is so
touchy, you never know...
Look in /var/log/dmesg for the scsi adapter logs.
# Software
What's your kernel version. Any patch applied ?
# Looking for a clue
You may try to profile the kernel on arrays with different numbers of
disks in raid 0 (to keep cpu out of equation at first).
It could help to find the bandwidth or the number of disks where the
actual speedup begins to fall behind the theoritical speedup and then
have the profiles at hand.
--
Lionel Bouton
-
"I wanted to be free, so I opensourced my whole DNA code" Gyver, 1999.