Dear software-RAID hackers and users,
I'm in the process of setting up RAID15 and RAID51. Right now, this is
for performance testing, but if operational characteristics are good,
it will go into production. The hardware platform is a dual-processor
P-III box (1Ghz processors), an ASUS CUR-DLS board, 2GB RAM, and two
3ware 8-channel IDE controllers (sitting on seperate PCI busses).
There are 16 disk drives, Maxtor 80GB beasts. The kernel is 2.4.17-rc2,
with the MD code compiled in (no modules). I use the raidtools2 package
of current Debian woody. As a testing platform, the machine does not
do anything besides the things described here:
In order to compare RAID15 to RAID51, I split the drives into two equal-sized
partitions, sd[a-p]2 for the RAID15, and sd[a-p]3 for the RAID51 tests.
The first layer was created yesterday: 8 mirror pairs, each pair with one
drive on each 3ware controller, rebuilt in parallel at a rate of about
7.5MB/s per mirror pair, 60MB/s aggregate. Next, two RAID5 sets were
created in parallel, each spanning the "3" partition of the 8 drives
of one, and the other, controller. This initial rebuild progressed
at about 12MB/s/set, 24MB/s aggregate. I waited until all this layer 1
rebuild was done.
So far, everything is well. I/O characteristics for sequential read and
write are OK on both variants. Sequential read from the RAID5 sets gives
me a whopping 150MB/s transfer speed. Great.
Then I started the second layer, making a RAID5 set out of the 8 mirror pairs
out of the "2" partitions, and making a mirror pair out of the two RAID5 sets
on the "3" partition. Technically, this seems to work, but the initial rebuild
runs extremely slow, compared to the above rates: the RAID15 rebuild goes
at about 380kB/s, and the RAID51 build at slightly over 100kB/s. I will report
on operational characteristics, once these rebuilds are through (in about
30 days, for the RAID51...)
Anybody got any explanation, hints, or even convincing discouragement?
best regards
Patrick
Please Cc: me on replies, I'm not subscribed to linux-kernel.
Here is the /etc/raidtab I am using. md[0-7] are the layer 1 mirror pairs,
md1[0-1] are the layer 1 RAID5 arrays, md8 is the RAID15 thing, and md9
is the RAID51 thing.
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sda2
raid-disk 0
device /dev/sdi2
raid-disk 1
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdb2
raid-disk 0
device /dev/sdj2
raid-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdc2
raid-disk 0
device /dev/sdk2
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdd2
raid-disk 0
device /dev/sdl2
raid-disk 1
raiddev /dev/md4
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sde2
raid-disk 0
device /dev/sdm2
raid-disk 1
raiddev /dev/md5
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdf2
raid-disk 0
device /dev/sdn2
raid-disk 1
raiddev /dev/md6
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdg2
raid-disk 0
device /dev/sdo2
raid-disk 1
raiddev /dev/md7
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdh2
raid-disk 0
device /dev/sdp2
raid-disk 1
raiddev /dev/md8
raid-level 5
nr-raid-disks 8
persistent-superblock 1
chunk-size 128
device /dev/md0
raid-disk 0
device /dev/md1
raid-disk 1
device /dev/md2
raid-disk 2
device /dev/md3
raid-disk 3
device /dev/md4
raid-disk 4
device /dev/md5
raid-disk 5
device /dev/md6
raid-disk 6
device /dev/md7
raid-disk 7
raiddev /dev/md9
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/md10
raid-disk 0
device /dev/md11
raid-disk 1
raiddev /dev/md10
raid-level 5
nr-raid-disks 8
persistent-superblock 1
chunk-size 32
device /dev/sda3
raid-disk 0
device /dev/sdb3
raid-disk 1
device /dev/sdc3
raid-disk 2
device /dev/sdd3
raid-disk 3
device /dev/sde3
raid-disk 4
device /dev/sdf3
raid-disk 5
device /dev/sdg3
raid-disk 6
device /dev/sdh3
raid-disk 7
raiddev /dev/md11
raid-level 5
nr-raid-disks 8
persistent-superblock 1
chunk-size 32
device /dev/sdi3
raid-disk 0
device /dev/sdj3
raid-disk 1
device /dev/sdk3
raid-disk 2
device /dev/sdl3
raid-disk 3
device /dev/sdm3
raid-disk 4
device /dev/sdn3
raid-disk 5
device /dev/sdo3
raid-disk 6
device /dev/sdp3
raid-disk 7
Finally, here's a current snapshot of /proc/mdstat:
Personalities : [raid0] [raid1] [raid5]
read_ahead 1024 sectors
md8 : active raid5 md7[7] md6[6] md5[5] md4[4] md3[3] md2[2] md1[1] md0[0]
272702080 blocks level 5, 128k chunk, algorithm 0 [8/8] [UUUUUUUU]
[========>............] resync = 43.9% (17120184/38957440) finish=909.8min speed=396K/sec
md9 : active raid1 md11[1] md10[0]
272814912 blocks [2/2] [UU]
[>....................] resync = 1.7% (4651736/272814912) finish=39435.7min speed=111K/sec
md11 : active raid5 sdp3[7] sdo3[6] sdn3[5] sdm3[4] sdl3[3] sdk3[2] sdj3[1] sdi3[0]
272814976 blocks level 5, 32k chunk, algorithm 0 [8/8] [UUUUUUUU]
md10 : active raid5 sdh3[7] sdg3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
272814976 blocks level 5, 32k chunk, algorithm 0 [8/8] [UUUUUUUU]
md0 : active raid1 sdi2[1] sda2[0]
38957504 blocks [2/2] [UU]
md1 : active raid1 sdj2[1] sdb2[0]
38957504 blocks [2/2] [UU]
md2 : active raid1 sdk2[1] sdc2[0]
38957504 blocks [2/2] [UU]
md3 : active raid1 sdl2[1] sdd2[0]
38957504 blocks [2/2] [UU]
md4 : active raid1 sdm2[1] sde2[0]
38957504 blocks [2/2] [UU]
md5 : active raid1 sdn2[1] sdf2[0]
38957504 blocks [2/2] [UU]
md6 : active raid1 sdo2[1] sdg2[0]
38957504 blocks [2/2] [UU]
md7 : active raid1 sdp2[1] sdh2[0]
38957504 blocks [2/2] [UU]
unused devices: <none>
On Saturday, 22 December 2001, at 10:38:32 +0100,
Patrick Schaaf wrote:
> Then I started the second layer, making a RAID5 set out of the 8 mirror pairs
> out of the "2" partitions, and making a mirror pair out of the two RAID5 sets
> on the "3" partition. Technically, this seems to work, but the initial rebuild
> runs extremely slow, compared to the above rates: the RAID15 rebuild goes
> at about 380kB/s, and the RAID51 build at slightly over 100kB/s. I will report
> on operational characteristics, once these rebuilds are through (in about
> 30 days, for the RAID51...)
>
There are two kernel tunnables that can modify RAID reconstruction speed:
/proc/sys/dev/raid/speed_limit_min
/proc/sys/dev/raid/speed_limit_max
They both are integer values in KB/s, the latter limiting the absolute
maximun RAID reconstruction speed (200 MB/s if I recall correctly, at
least for prior kernel versions). The former gives you the minimun
guaranteed reconstruction speed, and has a default (again, at least for
kernel kersions up to 2.4.7) of exactly 100 KB/s.
RAID reconstruction code is expected to only "access" disks when there
are no disk activity due to applications, with a guaranteed minimun of
"speed_limit_min". If this RAID code feels disks are busy due to
applications using the disks, reconstruction will take place at just
this minimun guaranteed speed (exactly 100 KB/s).
As far as I remember this happens with "layered" RAIDs, such as the
setup your are trying. To get a minimun reconstruction speed greater
than the default, just (for example):
echo "50000" > /proc/sys/dev/raid/speed_limit_min
Be careful not to give a too high value, because you will get a very
unresponsive system in case you set this parameter too high.
A couple of days ago Ingo Molnar submitted some patches for RAID in the
2.5.x series, with some improvements that maybe will find their way into
2.4.x. Check this threads:
http://marc.theaimsgroup.com/?t=100863779600002&r=1&w=2&n=5
http://marc.theaimsgroup.com/?l=linux-kernel&m=100876771508736&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=100876826309961&w=2
--
Jos? Luis Domingo L?pez
Linux Registered User #189436 Debian Linux Woody (P166 64 MB RAM)
jdomingo AT internautas DOT org => Spam at your own risk