Hallo!
This is a message originally sent to linux-scsi.
I got no reply so I think this was the wrong ML.
Please tell me if I should send more specific information about something.
Since january I struggle with this problem. It prevents me from running a backup server productively.
Thank you.
Lars
Hi there,
I have a problem with a SW-RAID6. It is reproduceable also after changing the hole hardware.
I startet with a Suse 11.2. The problem occured during writing much data to the array (high io load).
This is hopefully the right ML for my problem. Otherwise please excuse me and point me the the right ML.
Then I changed the PSU. Still errors on high load.
Then I changed the sata controller (Sil 3114 - sata_sil) with one with a different chipset (driver: sata_mv). Still errors on high load.
Then I changed the disk enclosure and all cables. Still errors.
Then I changed the mainboard (tyan opteron) with one from supermicro (H8SCM-F) with 6-core opteron. Still errors.
Then I changed to ubuntu 10.04 -> 10.10. Still errors
Then I tried different schedulars (noop,anticipatory,cfq,deadline). Still errors.
Then I tried kernel options: noapic + acpi=off without luck.
Then I changed the sata controller with a areca sas (driver: mvsas). Still errors.
Then I tried some different hdds (orig: Western Digital WDC WD2002FYPS + WDC WD2003FYYS; new: Seagate ST3320620NS). Still errors.
Then I tried some different kernel versions from ubuntu without luck:
2.6.32-22-server
2.6.35-25-server
Then I tried self compiled kernels without luck:
2.6.35.13
2.6.38.6
2.6.39: same problem occurs but later
The current configuration:
- tested only 64-bit kernels
- Supermicro H8SCM-F (AMD SR5650+SP5100) with 6-core opteron
- Areca (non-raid) ARC-1300ix-16 sas controller
- SW-RAID6 over 8 Western Digital HDDs (sone WDC WD2002FYPS + some WDC WD2003FYYS)
- redundant PSU
How to reproduce my problem:
mdadm -C /dev/md3 -l6 -n8 /dev/sd[c-h] missing missing
(the two missing hdds prevent this raid from initial sync)
Everything is just fine till yet.
Now produce high io-load:
mke2fs -j /dev/md3
The detailed history (search for Lars to get my posts):
https://bugs.launchpad.net/ubuntu/+bug/550559
The error messages changed a bit during the kernel versions.
The nearly complete dmesg output:
https://launchpadlibrarian.net/72325163/20110524.dmesg.out
Is there something I do wrong? Could someone help me to debug this?
Thanks
Lars
On Monday, June 06, 2011, Lars Täuber wrote:
>Hallo!
>
>This is a message originally sent to linux-scsi.
>I got no reply so I think this was the wrong ML.
>Please tell me if I should send more specific information about
>something. Since january I struggle with this problem. It prevents me
>from running a backup server productively.
>
>Thank you.
>Lars
>
>
>
>Hi there,
>
>I have a problem with a SW-RAID6. It is reproduceable also after changing
>the hole hardware. I startet with a Suse 11.2. The problem occured
>during writing much data to the array (high io load). This is hopefully
>the right ML for my problem. Otherwise please excuse me and point me the
>the right ML.
>
>
>Then I changed the PSU. Still errors on high load.
>Then I changed the sata controller (Sil 3114 - sata_sil) with one with a
>different chipset (driver: sata_mv). Still errors on high load. Then I
>changed the disk enclosure and all cables. Still errors.
>Then I changed the mainboard (tyan opteron) with one from supermicro
>(H8SCM-F) with 6-core opteron. Still errors. Then I changed to ubuntu
>10.04 -> 10.10. Still errors
>Then I tried different schedulars (noop,anticipatory,cfq,deadline). Still
>errors. Then I tried kernel options: noapic + acpi=off without luck.
>Then I changed the sata controller with a areca sas (driver: mvsas).
>Still errors. Then I tried some different hdds (orig: Western Digital
>WDC WD2002FYPS + WDC WD2003FYYS; new: Seagate ST3320620NS). Still
>errors. Then I tried some different kernel versions from ubuntu without
>luck: 2.6.32-22-server
>2.6.35-25-server
>
>Then I tried self compiled kernels without luck:
>2.6.35.13
>2.6.38.6
>2.6.39: same problem occurs but later
>
>The current configuration:
>- tested only 64-bit kernels
>- Supermicro H8SCM-F (AMD SR5650+SP5100) with 6-core opteron
>- Areca (non-raid) ARC-1300ix-16 sas controller
>- SW-RAID6 over 8 Western Digital HDDs (sone WDC WD2002FYPS + some WDC
>WD2003FYYS) - redundant PSU
>
>How to reproduce my problem:
>mdadm -C /dev/md3 -l6 -n8 /dev/sd[c-h] missing missing
>(the two missing hdds prevent this raid from initial sync)
>
>Everything is just fine till yet.
>Now produce high io-load:
>mke2fs -j /dev/md3
>
>The detailed history (search for Lars to get my posts):
>https://bugs.launchpad.net/ubuntu/+bug/550559
>
>The error messages changed a bit during the kernel versions.
>The nearly complete dmesg output:
>https://launchpadlibrarian.net/72325163/20110524.dmesg.out
>
>Is there something I do wrong? Could someone help me to debug this?
>Thanks
>Lars
Looking at your dmesg, I get the impression you have a bunch of disks that
are in need of a firmware update. Unforch, the dmesg snippet does not
include the drive discovery and identification data.
However, I would back that data up to another medium before I did that as I
had the seagate firmware update scramble the blkid's and partition names of
one of two 1Tb drives I have. Neither drive errors now, but the read/write
speeds for the 2nd identical drive are about 1/3rd the rate of the first.
Firmware updates are in the form of a bootable cd .iso, and you can
download the cd image from the makers site.
Cheers, gene
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Eisenhower!! Your mimeograph machine upsets my stomach!!
Hi Gene,
thanks for your fast reply.
Am Mon, 6 Jun 2011 05:59:02 -0400
Gene Heskett <[email protected]> schrieb:
> On Monday, June 06, 2011, Lars Täuber wrote:
[...]
> >
> >Is there something I do wrong? Could someone help me to debug this?
> >Thanks
> >Lars
>
> Looking at your dmesg, I get the impression you have a bunch of disks that
> are in need of a firmware update. Unforch, the dmesg snippet does not
> include the drive discovery and identification data.
Attached is the whole dmesg from a fresh boot up. (I hope I'm allowed to send this to the ML!?)
> However, I would back that data up to another medium before I did that as I
> had the seagate firmware update scramble the blkid's and partition names of
> one of two 1Tb drives I have. Neither drive errors now, but the read/write
> speeds for the 2nd identical drive are about 1/3rd the rate of the first.
There is no partition table nor any data on the disks, because I never could use the array.
> Firmware updates are in the form of a bootable cd .iso, and you can
> download the cd image from the makers site.
I didn't find any firmware updates from WD. How do they manage to have all the drivers possibly needed to talk to the disks?
Thanks again
Lars
> Cheers, gene
> --
On Monday, June 06, 2011, Lars Täuber wrote:
>Hi Gene,
>
>thanks for your fast reply.
>
>Am Mon, 6 Jun 2011 05:59:02 -0400
>
>Gene Heskett <[email protected]> schrieb:
>> On Monday, June 06, 2011, Lars Täuber wrote:
>[...]
>
>> >Is there something I do wrong? Could someone help me to debug this?
>> >Thanks
>> >Lars
>>
>> Looking at your dmesg, I get the impression you have a bunch of disks
>> that are in need of a firmware update. Unforch, the dmesg snippet
>> does not include the drive discovery and identification data.
>
>Attached is the whole dmesg from a fresh boot up. (I hope I'm allowed to
>send this to the ML!?)
>
>> However, I would back that data up to another medium before I did that
>> as I had the seagate firmware update scramble the blkid's and
>> partition names of one of two 1Tb drives I have. Neither drive errors
>> now, but the read/write speeds for the 2nd identical drive are about
>> 1/3rd the rate of the first.
>
>There is no partition table nor any data on the disks, because I never
>could use the array.
>
>> Firmware updates are in the form of a bootable cd .iso, and you can
>> download the cd image from the makers site.
>
>I didn't find any firmware updates from WD. How do they manage to have
>all the drivers possibly needed to talk to the disks?
>
>Thanks again
>Lars
>
I started here:
<http://support.wdc.com/product/download.asp?modelno=WD2003FYYS&x=8&y=10>
I entered your drives model number, and got here:
<http://community.wdc.com/t5/News-Announcements/bd-p/Announcements>
That is the link I got when I entered the WDC drive # shown in your dmesg.
That looks like too many choices, and probably is, so I would email the
support link shown and ask for clarification. Be sure to use the full
model number of the drive as shown on the drive or the packaging as its
possible I blew it with my copy/paste here.
The point is, the stuff is there. But its easier to find on Seagates site
I believe.
Cheers, gene
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Receiving a million dollars tax free will make you feel better than
being flat broke and having a stomach ache.
-- Dolph Sharp, "I'm O.K., You're Not So Hot"
Hi Gene,
Am Mon, 6 Jun 2011 09:20:32 -0400
Gene Heskett <[email protected]> schrieb:
[...]
> >
> I started here:
> <http://support.wdc.com/product/download.asp?modelno=WD2003FYYS&x=8&y=10>
>
> I entered your drives model number, and got here:
>
> <http://community.wdc.com/t5/News-Announcements/bd-p/Announcements>
>
> That is the link I got when I entered the WDC drive # shown in your dmesg.
> That looks like too many choices, and probably is, so I would email the
> support link shown and ask for clarification. Be sure to use the full
> model number of the drive as shown on the drive or the packaging as its
> possible I blew it with my copy/paste here.
I already found these, but this are all posts regarding consumer products. The harddisks I have are 2TB Raid Editions types. I couldn't find anything related there.
I sent an email to the WD support.
>
> The point is, the stuff is there. But its easier to find on Seagates site
> I believe.
The seagate drives were only for testing, because they are only 320GB in size.
I don't have six other drives for testing here, but I want to test with the same quantitiy of drives.
Is there anything else I can do OS-wise? Some hint how to debug deeper?
Thanks
Lars
Hi!
Am Mon, 6 Jun 2011 15:56:10 +0200
Lars Täuber <[email protected]> schrieb:
> I sent an email to the WD support.
WD told me that the firmware versions I use are the newest available for the
models. Here is a list of models in the array:
Model Serial # Firmware
WDC WD2002FYPS-01U1B1 WD-WCAVY4757773 04.05G05
WDC WD2002FYPS-02W3B0 WD-WCAVY5361793 04.01G01
WDC WD2002FYPS-02W3B0 WD-WCAVY6201161 04.01G01
WDC WD2002FYPS-02W3B0 WD-WCAVY6252449 04.01G01
WDC WD2002FYPS-02W3B0 WD-WCAVY6408178 04.01G01
WDC WD2003FYYS-02W0B0 WD-WMAY01123610 01.01D01
WDC WD2003FYYS-02W0B0 WD-WMAY01800979 01.01D01
WDC WD2003FYYS-02W0B0 WD-WMAY02037927 01.01D01
I think it really is a kernel problem because I had the same problem also
with seagate drives for which also no newer firmware seems to be available.
What can I do to track this down?
Is there someone that can confirm this problem on different hardware? It
really is simple to test if you have six drives and a (relatively fast)
running system.
Thanks
Lars
Hi there,
I just tested again with different accoustic managment settings (128 and 254)
in all disks.
The problem still remains.
Am I the only one using SW-RAID6 and having this problem?
I have several other SW-RAID6s productivly running. But they all have at
least 12 disks in an array and are very much older (HW + SW).
Newer running machines have at least 16 disks running (2.6.35).
Should I test older kernels than 2.6.32?
Please help me.
The box is ready to run all test you suggest me to run.
Thanks
Lars
On Wed, 08 Jun 2011, Lars T?uber wrote:
> Am I the only one using SW-RAID6 and having this problem?
Well, ask Arecca if these disks have problems with their RAID
controllers, and whether they can help you get hold of firmware updates
for the WD disks.
The WD RE4-GP 2TB might not be the only WD RE4 disk with quirky
firmware.
http://blog.fastmail.fm/2009/11/06/western-digital-re4-gp-2tb-drive-problems-and-solutions/
http://kb.lsi.com/KnowledgebaseArticle15592.aspx?Keywords=Re4+firmware
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
Hi Henrique,
thanks for your reply.
Am Wed, 8 Jun 2011 18:39:05 -0300
Henrique de Moraes Holschuh <[email protected]> schrieb:
> On Wed, 08 Jun 2011, Lars Täuber wrote:
> > Am I the only one using SW-RAID6 and having this problem?
>
> Well, ask Arecca if these disks have problems with their RAID
> controllers, and whether they can help you get hold of firmware updates
> for the WD disks.
>
> The WD RE4-GP 2TB might not be the only WD RE4 disk with quirky
> firmware.
>
> http://blog.fastmail.fm/2009/11/06/western-digital-re4-gp-2tb-drive-problems-and-solutions/
> http://kb.lsi.com/KnowledgebaseArticle15592.aspx?Keywords=Re4+firmware
According to this thread:
http://community.wdc.com/t5/Other-Internal-Drives/WD2002FYPS-firmware/td-p/63330
the WD2002FYPS-02W3B0 models seem to be really bad for RAID systems.
The former models with the most recent firmware seem to work somehow:
Model: WD2002FYPS-01U1B1
FW: 04.05G05
Seems I have to exchange the drives to get a working array.
I got an reply from Western Digital that all my disks have the most recent
firmware available.
I wonder why I had the very same problem with Seagate Barracuda ES 320GB
drives!
Thanks again and sunny greatings to brazil.
Lars