Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
(32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
seeing about 216-224 MB/S total throughput on writes to local
arrays on 2.4.18.
The system is also running an Intel Gigabit Ethernet Card at
116-122 MB/S with full network traffic and writing this traffic to
the 3Ware arrays. All this hardware is running on a Serverworks
HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
processors.
After running for about 3 hours, the system will hard hang and die.
Using debugging tools, I have isolated to the hang to the 3Ware
adapters. If I remove all but a single 3Ware adapter, the system will
run reliably for days at these data rates. The moment I add more
than one 3Ware 7810 adapter, the system will lock up. Recent testing
reveals that the hang is in the 3Ware card itself (all the LEDs go
on at once and stay on). Attempts by the system to reset the adapter
fail until the system is power cycled.
3Ware dfriver version is .16 from the 2.4.18 tree. Firmware is the 48 bit
LBA version.
Please advise.
Jeff
On Wed, Feb 27, 2002 at 10:25:45AM -0700, Jeff V. Merkey wrote:
>
More info. I put in some trace code to determine how many io buffer
heads were being fed to each adapter. When the number of bh's reaches
numbers above 4244+- buffer heads outstanding at one time, then I see the
cards lockup.
Jeff
>
>
> Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> seeing about 216-224 MB/S total throughput on writes to local
> arrays on 2.4.18.
>
> The system is also running an Intel Gigabit Ethernet Card at
> 116-122 MB/S with full network traffic and writing this traffic to
> the 3Ware arrays. All this hardware is running on a Serverworks
> HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> processors.
>
> After running for about 3 hours, the system will hard hang and die.
> Using debugging tools, I have isolated to the hang to the 3Ware
> adapters. If I remove all but a single 3Ware adapter, the system will
> run reliably for days at these data rates. The moment I add more
> than one 3Ware 7810 adapter, the system will lock up. Recent testing
> reveals that the hang is in the 3Ware card itself (all the LEDs go
> on at once and stay on). Attempts by the system to reset the adapter
> fail until the system is power cycled.
>
> 3Ware dfriver version is .16 from the 2.4.18 tree. Firmware is the 48 bit
> LBA version.
>
> Please advise.
>
> Jeff
On Feb 27, 2002 10:25 -0700, Jeff V. Merkey wrote:
> Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> seeing about 216-224 MB/S total throughput on writes to local
> arrays on 2.4.18.
Have you done any kind of variations on this configuration to see when
or where the maximum throughput happens? Daniel and I were speculating
about where the 3ware limits are. Specs say 100MB/s per adapter (for
both 6000 and 7000 series), you would probably hit max bandwidth with
2 adapters. The drives themselves are not a limiting factor, unless
you are down to striping across only 2 drives instead of all 8. I take
it you are using the hardware RAID instead of software MD RAID?
> The system is also running an Intel Gigabit Ethernet Card at
> 116-122 MB/S with full network traffic and writing this traffic to
> the 3Ware arrays. All this hardware is running on a Serverworks
> HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> processors.
Does this board have multiple PCI busses? Is the GigE card on a
different bus than the 3ware cards?
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
On Wed, Feb 27, 2002 at 10:48:25AM -0700, Andreas Dilger wrote:
> On Feb 27, 2002 10:25 -0700, Jeff V. Merkey wrote:
> > Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> > for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> > (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> > seeing about 216-224 MB/S total throughput on writes to local
> > arrays on 2.4.18.
>
> Have you done any kind of variations on this configuration to see when
> or where the maximum throughput happens? Daniel and I were speculating
> about where the 3ware limits are. Specs say 100MB/s per adapter (for
> both 6000 and 7000 series), you would probably hit max bandwidth with
> 2 adapters. The drives themselves are not a limiting factor, unless
> you are down to striping across only 2 drives instead of all 8. I take
> it you are using the hardware RAID instead of software MD RAID?
Single adapter can push 130 MB/S with 8 drives striped at RAID 0
(hardware RAID not software). With 2 adapters with 8 drives, the
max rate of 216 MB/S is achieved. Adding additional adapters
on the same PCI bus hits the wall at 238+- MB/S. I have not tried
putting the two other 3Ware's on the 66 Mhz bus, but in theory, this
would result in 350 MB/S total throughput from what I am seeing
with the Gigabit ethernet adapter.
>
> > The system is also running an Intel Gigabit Ethernet Card at
> > 116-122 MB/S with full network traffic and writing this traffic to
> > the 3Ware arrays. All this hardware is running on a Serverworks
> > HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> > processors.
>
> Does this board have multiple PCI busses? Is the GigE card on a
> different bus than the 3ware cards?
The Gigabit adapter is running on the 66 Mhz bus. When combined with
SCI I am able to push 238 MB/S through the SCI to the 3Ware adapters
and out to disk. At present, I have modified the SCI drivers and I am
DMA'ing directly into Linus buffer cache from the SCI disk data.
Very very fast. The 3Ware adapters are barfing when the BH count
gets too high. I have no idea why.
Jeff
>
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/