Each PCI-e x1 card has 1 veliciraptor on it now.
Got an x4 card wit 4 sata ports:
Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
but pretty close!
(For my RAID5)
Previously my write was limited to 400-420MiB/s now I see an additional
120-125 MiB/s increase!
jpiszcz@p34:/x/f$ dd if=/dev/zero of=bigfile bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 20.7054 s, 519 MB/s
jpiszcz@p34:/x/f$ sync
jpiszcz@p34:/x/f$ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 20.4973 s, 524 MB/s
jpiszcz@p34:/x/f$ sync
jpiszcz@p34:/x/f$ dd if=bigfile of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 11.3529 s, 946 MB/s
jpiszcz@p34:/x/f$ sync
jpiszcz@p34:/x/f$ dd if=bigfile.1 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 11.2635 s, 953 MB/s
jpiszcz@p34:/x/f$
--
For all disks:
Something I noticed is the x1 PCI-e cards are doing around 68MiB/s each
for 3 of them where the x4 has no issue pumping out 100MiB/s+ without a
problem, however keep in mind the bus is probably already taxed from the
6 sata drives on the southbridge.
vmstat output:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
1 VR
0 1 160 45220 341772 6480468 0 0 122112 0 584 2082 1 7 73 20
0 1 160 46592 455436 6362088 0 0 113664 0 495 1968 0 4 74 21
2 VR
1 1 160 45540 3027724 3720340 0 0 243216 0 1006 4030 0 9 74 17
0 2 160 44988 3262220 3480648 0 0 234480 0 1008 4134 0 8 73 19
3 VR
1 2 160 44816 6600068 50476 0 0 330248 16 1342 4126 0 12 70 18
0 3 160 45440 6599812 50264 0 0 316032 8 1296 3878 0 12 72 17
4 VR
0 4 160 44504 6602488 47644 0 0 495232 0 1992 6081 0 20 57 23
1 3 160 45500 6602796 45980 0 0 483968 0 1915 6207 0 20 54 26
5 VR
1 5 160 43932 6602972 45304 0 0 606080 0 2375 6622 0 25 56 19
1 4 160 45412 6601852 45160 0 0 618756 0 2431 6791 0 25 53 21
6 VR
0 6 160 45000 6602348 44512 0 0 683904 8 2746 7880 0 31 42 27
0 6 160 45248 6602028 44460 0 0 705792 0 2754 7564 0 31 45 24
7 VR
2 6 160 46744 6599020 44688 0 0 748204 17 3042 9084 0 34 40 26
3 6 160 46592 6598824 44372 0 0 747520 8 2975 9047 1 33 31 36
8 VR
2 7 160 46512 6598612 44580 0 0 761184 16 3089 9937 0 36 40 24
2 7 160 44528 6600392 44360 0 0 759720 8 2993 9522 0 36 36 28
9 VR
2 8 160 47152 6596824 44572 0 0 767016 0 3075 9730 1 37 39 24
2 7 160 46576 6597728 44688 0 0 771200 0 3032 9568 0 37 40 23
10 VR
0 10 160 45048 6598240 44428 0 0 889072 8 3599 11561 0 47 20 33
2 10 160 45232 6598116 44772 0 0 890112 0 3495 11547 0 46 23 31
11 VR
4 8 160 45536 6594716 44600 0 0 996352 0 3947 12134 1 62 13 25
2 9 160 45348 6594912 44096 0 0 1009152 0 3949 11949 0 63 10 28
12 VR
6 8 160 45092 6583136 47016 0 0 1063200 0 4187 12394 1 71 9 21
3 11 160 47080 6578492 47588 0 0 1058412 0 4224 12547 1 72 8 20
Just about 1 gigabyte per second total aggregate read for all drives on a
965 chipset!
Justin.
On Mon, 7 Jul 2008, Justin Piszcz wrote:
> Each PCI-e x1 card has 1 veliciraptor on it now.
> Got an x4 card wit 4 sata ports:
>
> Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
> but pretty close!
Going to remove one of the drives from the x1 card and put it on the x4
card instead, then I will use all 4 SATA ports on the x4 and hopefully get
better bw.
If you look at 7,8,9 there is little improvement: (PCI-e x1)
7 VR
2 6 160 46744 6599020 44688 0 0 748204 17 3042 9084 0 34 40 26
3 6 160 46592 6598824 44372 0 0 747520 8 2975 9047 1 33 31 36
8 VR
2 7 160 46512 6598612 44580 0 0 761184 16 3089 9937 0 36 40 24
2 7 160 44528 6600392 44360 0 0 759720 8 2993 9522 0 36 36 28
9 VR
2 8 160 47152 6596824 44572 0 0 767016 0 3075 9730 1 37 39 24
2 7 160 46576 6597728 44688 0 0 771200 0 3032 9568 0 37 40 23
But once I hit the drives on the x4 card, vroom vroom!
10 VR
0 10 160 45048 6598240 44428 0 0 889072 8 3599 11561 0 47 20 33
2 10 160 45232 6598116 44772 0 0 890112 0 3495 11547 0 46 23 31
11 VR
4 8 160 45536 6594716 44600 0 0 996352 0 3947 12134 1 62 13 25
2 9 160 45348 6594912 44096 0 0 1009152 0 3949 11949 0 63 10 28
12 VR
6 8 160 45092 6583136 47016 0 0 1063200 0 4187 12394 1 71 9 21
3 11 160 47080 6578492 47588 0 0 1058412 0 4224 12547 1 72 8 20
Justin.
On Mon, 7 Jul 2008, Justin Piszcz wrote:
>
>
> On Mon, 7 Jul 2008, Justin Piszcz wrote:
>
>> Each PCI-e x1 card has 1 veliciraptor on it now.
>> Got an x4 card wit 4 sata ports:
>>
>> Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
>> but pretty close!
>
> Going to remove one of the drives from the x1 card and put it on the x4
> card instead, then I will use all 4 SATA ports on the x4 and hopefully get
> better bw.
>
Four drives on the x4 card, MAX bandwidth for every disk.
p34:~# dd if=/dev/sdi of=/dev/null bs=1M &
[1] 4720
p34:~# dd if=/dev/sdj of=/dev/null bs=1M &
[2] 4721
p34:~# dd if=/dev/sdk of=/dev/null bs=1M &
[3] 4722
p34:~# dd if=/dev/sdl of=/dev/null bs=1M &
[4] 4723
p34:~#
120MiB/s per each one!
Re-running dd test with all 12 disks:
1.1 gigabytes per second read!
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 120 59104 6632220 52228 0 0 0 40 168 517 0 0 100 0
0 0 120 59104 6632220 52228 0 0 0 0 20 291 0 0 100 0
3 10 120 43516 6635576 51924 0 0 1051776 62 4221 12301 1 70 11 19
6 9 160 44420 6634720 51788 0 0 1117284 0 4435 12308 1 75 5 19
6 9 160 47436 6631100 51676 0 0 1110300 0 4449 11438 1 76 3 20
2 10 160 46740 6632048 51948 0 0 1137920 0 4447 12251 1 75 8 17
9 7 160 45248 6632056 52004 0 8 1127940 45 4559 13259 1 74 9 17
3 9 160 44152 6634780 49960 0 0 1132032 12 4471 12962 0 75 8 16
4 9 160 44160 6634960 49380 0 0 1129216 8 4430 12545 0 76 7 16
After:
About the same for write:
$ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 20.4056 s, 526 MB/s
'nuff said for read :)
$ dd if=bigfile.1 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 10.2841 s, 1.0 GB/s
Justin.
On Mon, 7 Jul 2008, Justin Piszcz wrote:
>
>
> On Mon, 7 Jul 2008, Justin Piszcz wrote:
>
>>
>>
>> On Mon, 7 Jul 2008, Justin Piszcz wrote:
>>
>>> Each PCI-e x1 card has 1 veliciraptor on it now.
>>> Got an x4 card wit 4 sata ports:
>>>
>>> Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
>>> but pretty close!
>>
>> Going to remove one of the drives from the x1 card and put it on the x4
>> card instead, then I will use all 4 SATA ports on the x4 and hopefully get
>> better bw.
>>
>
> Four drives on the x4 card, MAX bandwidth for every disk.
>
> p34:~# dd if=/dev/sdi of=/dev/null bs=1M &
> [1] 4720
> p34:~# dd if=/dev/sdj of=/dev/null bs=1M &
> [2] 4721
> p34:~# dd if=/dev/sdk of=/dev/null bs=1M &
> [3] 4722
> p34:~# dd if=/dev/sdl of=/dev/null bs=1M &
> [4] 4723
> p34:~#
>
> 120MiB/s per each one!
>
> Re-running dd test with all 12 disks:
>
> 1.1 gigabytes per second read!
>
bonnie++ benchmarks:
http://home.comcast.net/~jpiszcz/20080707/veliciraptors_with_x4.html
Justin Piszcz wrote:
> Each PCI-e x1 card has 1 veliciraptor on it now.
> Got an x4 card wit 4 sata ports:
Useful - which card?
David
On Tue, 8 Jul 2008, David Greaves wrote:
> Justin Piszcz wrote:
>> Each PCI-e x1 card has 1 veliciraptor on it now.
>> Got an x4 card wit 4 sata ports:
>
> Useful - which card?
>
> David
>
>
StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
Justin.
On Tue, 8 Jul 2008, Justin Piszcz wrote:
>
>
> On Tue, 8 Jul 2008, David Greaves wrote:
>
>> Justin Piszcz wrote:
>>> Each PCI-e x1 card has 1 veliciraptor on it now.
>>> Got an x4 card wit 4 sata ports:
>>
>> Useful - which card?
>>
>> David
>>
>>
>
> StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
>
> Justin.
>
>
>
Chipset: Marvell 88SX7042
SATA Connectivity: Use four internal ports at the same time or two
internal and two external ports
Which is fully supported in the latest kernels (didn't try an old kernel):
linux-2.6.25.10/drivers/ata/sata_mv.c
/* Marvell 7042 support */
{ PCI_VDEVICE(MARVELL, 0x7042), chip_7042 },
On Tue, Jul 08, 2008 at 04:39:34AM -0400, Justin Piszcz wrote:
> On Tue, 8 Jul 2008, Justin Piszcz wrote:
>> On Tue, 8 Jul 2008, David Greaves wrote:
>>> Justin Piszcz wrote:
>>>> Each PCI-e x1 card has 1 veliciraptor on it now.
>>>> Got an x4 card wit 4 sata ports:
>>> Useful - which card?
>>
>> StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
>>
>>
> Chipset: Marvell 88SX7042
> SATA Connectivity: Use four internal ports at the same time or two internal
> and two external ports
>
> Which is fully supported in the latest kernels (didn't try an old kernel):
>
> linux-2.6.25.10/drivers/ata/sata_mv.c
>
> /* Marvell 7042 support */
> { PCI_VDEVICE(MARVELL, 0x7042), chip_7042 },
How have that card and the array attached to it been doing since you
originally posted this?
Doesn't that HighPoint 23xx card also use the Marvell 88SX7042 chip?
I remember seeing the threads about the HighPoint card silently
corrupting data by silently writing over parts of the disk(s) with
its own info. I'm guessing that's not a "feature" of the Marvell
chip, but still... admittedly irrational fear here :)
It would be nice if that Supermicro 8-port SATA card (AOC-SAT2-MV8)
was available in PCIe... I wonder if it's possible to just graft
multiple SiI 3132 controllers (2 SATA each) on a single PCIe card?
On Thu, 31 Jul 2008, Matt Garman wrote:
> On Tue, Jul 08, 2008 at 04:39:34AM -0400, Justin Piszcz wrote:
>> On Tue, 8 Jul 2008, Justin Piszcz wrote:
>>> On Tue, 8 Jul 2008, David Greaves wrote:
>>>> Justin Piszcz wrote:
>>>>> Each PCI-e x1 card has 1 veliciraptor on it now.
>>>>> Got an x4 card wit 4 sata ports:
>>>> Useful - which card?
>>>
>>> StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
>>>
>>>
>> Chipset: Marvell 88SX7042
>> SATA Connectivity: Use four internal ports at the same time or two internal
>> and two external ports
>>
>> Which is fully supported in the latest kernels (didn't try an old kernel):
>>
>> linux-2.6.25.10/drivers/ata/sata_mv.c
>>
>> /* Marvell 7042 support */
>> { PCI_VDEVICE(MARVELL, 0x7042), chip_7042 },
>
> How have that card and the array attached to it been doing since you
> originally posted this?
No problems to report yet, working well, I wish I had multiple x4 or x16
slots and use those instead of x1 cards (the mobo would need to support
the I/O as well)..
>
> Doesn't that HighPoint 23xx card also use the Marvell 88SX7042 chip?
I read remeber 'locked' 64kb read problem but not any corruption issue.
> I remember seeing the threads about the HighPoint card silently
> corrupting data by silently writing over parts of the disk(s) with
Also I MD5SUM -c (verify) 100-150GiB of data daily or so to make sure it
is OK, so far, no problems.
> its own info. I'm guessing that's not a "feature" of the Marvell
> chip, but still... admittedly irrational fear here :)
Have not seen it yet.
>
> It would be nice if that Supermicro 8-port SATA card (AOC-SAT2-MV8)
> was available in PCIe... I wonder if it's possible to just graft
> multiple SiI 3132 controllers (2 SATA each) on a single PCIe card?
Yes, I asked Supermicro about it and they never mailed me back about a
PCIe version..
That's a good question, I suppose if it was easy it would have already
been done?
Justin.