What are the recommended software RAID patch(es) for 2.2.20.
I have applied, but not yet tested, raid-2.2.19-A1, which
fails on one hunk in init.c which I think is unimportant
(seemingly 2.2.20 has a full boot line to device translation
table without conditionality of compilation).
Do I need to apply anything else? (-A2, -A3, -B1, -B2 or
whatever) in which case where do I get them from? (not
kernel.org/people/mingo or wherever, apparently).
--
Alex Bligh
On Thu, Nov 22, 2001 at 06:52:24PM -0000, you [Alex Bligh - linux-kernel] claimed:
> What are the recommended software RAID patch(es) for 2.2.20.
>
> I have applied, but not yet tested, raid-2.2.19-A1, which
> fails on one hunk in init.c which I think is unimportant
> (seemingly 2.2.20 has a full boot line to device translation
> table without conditionality of compilation).
I just applied raid-2.2.19-A1 on 2.2.20 and have it running.
I'm not 100% sure it works, though, since I get always the same md5sum from
cat /dev/hde | md5sum
or
cat /dev/hdg | md5sum
(hdg and hde are not the same, but successive hde runs are the same as are
hdg runs.)
but
cat /dev/md0 | md5sum
is different every time. md0 consists of hde and hdg striped together.
(Again, I'm not expecting it to be same as hde or hdg, but consistent over
successive runs.) Successive hde and hdg runs are consistent even when run
in parallel.
Any ideas what could cause this? A short read or something else trivial?
I'll try to go back to 2.2.18pre19 soon, and see if it happens there. Also,
I'll try to see where the difference is. The first GB of md0 md5sums ok.
I'm having a lot of problem with this setup, and raid code is not my primary
suspect. My gut feeling is that you should be safe with raid-2.2.19-A1 on
top of 2.2.20 and my problem is elsewhere.
> Do I need to apply anything else? (-A2, -A3, -B1, -B2 or
> whatever)
I reckon not. -B3, -A1 etc are version codes Ingo likes to use.
-- v --
[email protected]
On Thu, Nov 22, 2001 at 09:05:03PM +0200, you [Ville Herva] claimed:
>
> cat /dev/hde | md5sum
> or
> cat /dev/hdg | md5sum
> (hdg and hde are not the same, but successive hde runs are the same as are
> hdg runs.)
>
> but
> cat /dev/md0 | md5sum
>
> is different every time. md0 consists of hde and hdg striped together.
> (Again, I'm not expecting it to be same as hde or hdg, but consistent over
> successive runs.) Successive hde and hdg runs are consistent even when run
> in parallel.
>
> Any ideas what could cause this? A short read or something else trivial?
>
> I'll try to go back to 2.2.18pre19 soon, and see if it happens there. Also,
> I'll try to see where the difference is. The first GB of md0 md5sums ok.
Ummh. I'm doing successive runs for /dev/md0 GB at a time, and it seems to
get corrupted in the middle:
#!/usr/bin/perl
for ($i = 0; $i < 75; $i++)
{
$block = 1024**2;
$count = 1024;
$| = 1;
print "At $i GB\n";
system "(dd if=/dev/md0 bs=$block count=$count skip=".($i*$count).
"| md5sum) 2>&1";
}
diff -c on (modified) output:
*** 5,11 ****
At 4 GB 131e2916f3155f7c6df63fe2257e0350 -
At 5 GB 502be9c039744eb761f89ada152a1745 -
At 6 GB 07012ffe77ad7d6565f2e9576f1cf91e -
! At 7 GB ffa5545ee518d3a7724831012d3e4c44 -
At 8 GB eec233bf66d33fc81bfa895b022c4b04 -
At 9 GB c62e78b9401f91199ce558242faf5da5 -
At 10 GB f41d004b63c2481245320c28b9366b08 -
--- 5,11 ----
At 4 GB 131e2916f3155f7c6df63fe2257e0350 -
At 5 GB 502be9c039744eb761f89ada152a1745 -
At 6 GB 07012ffe77ad7d6565f2e9576f1cf91e -
! At 7 GB 4bbbaeefe786c760e342342f6a85ad4e -
At 8 GB eec233bf66d33fc81bfa895b022c4b04 -
At 9 GB c62e78b9401f91199ce558242faf5da5 -
At 10 GB f41d004b63c2481245320c28b9366b08 -
(... still running)
I'm puzzled as to why this happens, since (as said) hde and hdg both md5sum
quite ok. Raid stuff, I gather, should be fairly solid, no?
-- v --
[email protected]
On Thu, Nov 22, 2001 at 09:54:24PM +0200, you [Ville Herva] claimed:
>
> Ummh. I'm doing successive runs for /dev/md0 GB at a time, and it seems to
> get corrupted in the middle:
>
> #!/usr/bin/perl
> for ($i = 0; $i < 75; $i++)
> {
> $block = 1024**2;
> $count = 1024;
> $| = 1;
> print "At $i GB\n";
>
> system "(dd if=/dev/md0 bs=$block count=$count skip=".($i*$count).
> "| md5sum) 2>&1";
> }
>
> diff -c on (modified) output:
>
> *** 5,11 ****
> At 4 GB 131e2916f3155f7c6df63fe2257e0350 -
> At 5 GB 502be9c039744eb761f89ada152a1745 -
> At 6 GB 07012ffe77ad7d6565f2e9576f1cf91e -
> ! At 7 GB ffa5545ee518d3a7724831012d3e4c44 -
> At 8 GB eec233bf66d33fc81bfa895b022c4b04 -
> At 9 GB c62e78b9401f91199ce558242faf5da5 -
> At 10 GB f41d004b63c2481245320c28b9366b08 -
> --- 5,11 ----
> At 4 GB 131e2916f3155f7c6df63fe2257e0350 -
> At 5 GB 502be9c039744eb761f89ada152a1745 -
> At 6 GB 07012ffe77ad7d6565f2e9576f1cf91e -
> ! At 7 GB 4bbbaeefe786c760e342342f6a85ad4e -
> At 8 GB eec233bf66d33fc81bfa895b022c4b04 -
> At 9 GB c62e78b9401f91199ce558242faf5da5 -
> At 10 GB f41d004b63c2481245320c28b9366b08 -
> (... still running)
Ok, after 5 runs, it turned out that the error is always at 7GB or 47GB. hde
or hdg never fail when run separately (in parallel or not). This is propably
an IDE problem anyhow, since:
Nov 22 16:50:31 linux kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Nov 22 16:50:31 linux kernel: hde: dma_intr: error=0x84 { DriveStatusError BadCRC }
Nov 22 22:47:11 linux kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Nov 22 22:47:11 linux kernel: hde: dma_intr: error=0x84 { DriveStatusError BadCRC }
Nov 23 02:48:30 linux kernel: hdg: 24 bytes in FIFO
Nov 23 02:48:30 linux kernel: hdg: ide_dma_timeout: Lets do it again!stat = 0x58 , dma_stat = 0x20
Nov 23 02:48:30 linux kernel: hdg: DMA disabled
Nov 23 02:48:30 linux kernel: hdg: timeout waiting for DMA
Nov 23 02:48:30 linux kernel: hdg: 0 bytes in FIFO
Nov 23 02:48:30 linux kernel: hdg: ide_dma_timeout: Lets do it again!stat = 0x80, dma_stat = 0x20
Nov 23 02:48:30 linux kernel: hdg: DMA disabled
Nov 23 02:48:30 linux kernel: hdg: irq timeout: status=0x80 { Busy }
Nov 23 02:48:30 linux kernel: hdg: DMA disabled
Nov 23 02:48:30 linux kernel: hdg: ide_set_handler: handler not null; old=c017fd
(This was with 2.2.20+IDE 05042001 + Tim Hockin's hpt366.c patch).
It went back to 2.2.18pre19 and the md5sum of md0 are consistent. I saw a
couple of BadCRC errors, however.
I'm trying again with 2.2.20/IDE 05042001/no hpt patch/slightly older
bios/UDMA33 instead of UDMA66. (Just _too_ many variables... Phew.)
The controller is HPT370A (KT7A-RAID) and drives(hde,hdg) are IBM-DPTA-373420
(32.5GB).
-- v --
[email protected]
On Fri, Nov 23, 2001 at 02:35:02PM +0200, you [Ville Herva] claimed:
> On Thu, Nov 22, 2001 at 09:54:24PM +0200, you [Ville Herva] claimed:
>
> I'm trying again with 2.2.20/IDE 05042001/no hpt patch/slightly older
> bios/UDMA33 instead of UDMA66. (Just _too_ many variables... Phew.)
2.2.20/IDE 05042001/no hpt patch/UDM33 gives no IDE errors, but
/dev/md0 md5sums still differ.
Since I didn't see the corruption with /dev/hde nor /dev/hdg, I'm beginning
to wonder if this is in fact RAID/VM/some other problem...?
2.2.19 included the VM rewrite, and I also patched the 2.2.20 kernel with
Andrea's VM-locked-1 -patch (which I figure shouldn't really affect read
path?)
The 2.2.18pre19 kernel (that works) uses 2.2.17-raid A0-patch, and
2.2.20 uses 2.2.19-A1. The differences between the two seems trivial.
I would be delighted if someone could confirm that 2.2.20+raid-A1 works 100%
correct... Just md5summing the contents of /dev/md0 3-4 times should do if
were lucky. It takes time, though...
-- v --
[email protected]
On Fri, Nov 23, 2001 at 04:39:32PM +0200, you [Ville Herva] claimed:
> On Fri, Nov 23, 2001 at 02:35:02PM +0200, you [Ville Herva] claimed:
> > On Thu, Nov 22, 2001 at 09:54:24PM +0200, you [Ville Herva] claimed:
> >
> > I'm trying again with 2.2.20/IDE 05042001/no hpt patch/slightly older
> > bios/UDMA33 instead of UDMA66. (Just _too_ many variables... Phew.)
>
> 2.2.20/IDE 05042001/no hpt patch/UDM33 gives no IDE errors, but
> /dev/md0 md5sums still differ.
>
> Since I didn't see the corruption with /dev/hde nor /dev/hdg, I'm beginning
> to wonder if this is in fact RAID/VM/some other problem...?
>
> 2.2.19 included the VM rewrite, and I also patched the 2.2.20 kernel with
> Andrea's VM-locked-1 -patch (which I figure shouldn't really affect read
> path?)
>
> The 2.2.18pre19 kernel (that works) uses 2.2.17-raid A0-patch, and
> 2.2.20 uses 2.2.19-A1. The differences between the two seems trivial.
>
> I would be delighted if someone could confirm that 2.2.20+raid-A1 works 100%
> correct... Just md5summing the contents of /dev/md0 3-4 times should do if
> were lucky. It takes time, though...
Ok, it isn't raid. It took a little more effort to reproduce with plain
/dev/hd[eg] read, but the md5sum mismatch eventually happaned.
This was with 2.2.20+IDE, no hpt patch. The drives were forced to UDMA33 as
opposed to (default) UDMA66. This time, nothing shows up in the log.
I'll try again with 2.2.18pre19 (this time harder) to make sure it really
_really_ doesn't happen with it.
Then I'll give 2.4.15 a shot to see if hpt behaves (2.4 lacks e2compr
however, so I'm not sure I can actually use it).
-- v --
[email protected]
On Fri, Nov 23, 2001 at 11:32:11PM +0200, you [Ville Herva] claimed:
>
> Ok, it isn't raid. It took a little more effort to reproduce with plain
> /dev/hd[eg] read, but the md5sum mismatch eventually happaned.
>
> This was with 2.2.20+IDE, no hpt patch. The drives were forced to UDMA33 as
> opposed to (default) UDMA66. This time, nothing shows up in the log.
>
> I'll try again with 2.2.18pre19 (this time harder) to make sure it really
> _really_ doesn't happen with it.
(I ahven't gotten around to try this, but...)
> Then I'll give 2.4.15 a shot to see if hpt behaves (2.4 lacks e2compr
> however, so I'm not sure I can actually use it).
...I compiled 2.4.15 + Tim Hockin's HPT370 patch. I let it run the test over
night, but
dd if=/dev/md0 bs=1048576 skip=65536 count=1024 | md5sum
had stalled at _second_ run. The first, identical run had gone through.
The point where it happened is after dd has read the last (not whole GB) and
is trying to skip slightly more than the size of md0:
At 63 GB
756+1 records in
756+1 records out
b1a566b4b54905ab21d138670b7abae3 -
At 64 GB
<stalls>
This never happened with 2.2.*
What's more, the md5sum of 1st and 2nd run differ at 31GB:
@@ -125,7 +125,7 @@
At 31 GB
1024+0 records in
1024+0 records out
-c137bbbc546b03b54188697db0758143 -
+0bee73556a9fbb38d3af79b0d11b18ec -
At 32 GB
1024+0 records in
1024+0 records out
There's two BadCRC errors in the log:
hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hdg: dma_intr: error=0x84 { DriveStatusError BadCRC }
hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hdg: dma_intr: error=0x84 { DriveStatusError BadCRC }
Bleah. I'll try 2.4.15 vanilla (no hpt patch) and UDMA33 (instead of UDMA66)
next.
-- v --
[email protected]
On Sat, Nov 24, 2001 at 08:52:01AM +0200, you [Ville Herva] claimed:
>
> had stalled at _second_ run. The first, identical run had gone through.
> The point where it happened is after dd has read the last (not whole GB) and
> is trying to skip slightly more than the size of md0:
>
> At 63 GB
> 756+1 records in
> 756+1 records out
> b1a566b4b54905ab21d138670b7abae3 -
> At 64 GB
> <stalls>
>
> This never happened with 2.2.*
The "stall" seems to be a dd bug:
strace dd if=/dev/md0 of=/dev/null bs=1048576 count=1024 skip=65536
(...)
open("/dev/md0", O_RDONLY|O_LARGEFILE) = 0
(...)
_llseek(0, 68719476736, 0xbffff2d0, SEEK_SET) = -1 EINVAL (Invalid argument)
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
1048576) =
1048576
read(0, "\366&\3704\327\310\243\232\332\220\223\235\367\206\300"...,
1048576) =
1048576
read(0, "n\3164\326\366>\330\5\377\236\303>Vf\377f\317<\226\212"...,
1048576) =
1048576
(...)
(reads propably the whole /dev/md0)
...back to investigating the differing md5sums...
-- v --
[email protected]
Summary of the thread so far:
- I had differing md5sum when reading /dev/md0 several time
under 2.2.20 + idepatch
- hw involved is 2x IBM-DPTA-373420 (32.5GB), both on their own
IDE channel as the sole device.
- HPT370 on Abit KT7A-RAID
- Duron 900
- the drives default to UDMA66
- several BIOSes tried
- I managed to reproduce the md5sum failures reading /dev/hde and /dev/hdg
although it was harder to trigger.
- It happened on 2.5.15 + Hockins' hpt patch as well
- It happened on 2.2.20+IDE with and without Hockin's patch
- It happened on 2.2.20+IDE even at UDMA33
- The thread starts here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=100645594530137&w=2
I managed to reproduce the corruption on 2.2.18pre19, UDMA66 as well. It
seems harder to trigger (took 9 runs), but eventually failed 10GB blocks out
of 64. (other kernels usually failed 0-2 GB blocks per run.)
What's interesting is that even though 2.2.18pre19 and 2.2.20 both default
to UDMA66, 2.2.20 manages to pull 30-35MB/s off md0, whereas 2.2.18pre19
only gets ~25MB/s. I'm not sure why that is (different VM, tuning of
elevator, newer IDE patch, whatever), but that only could explain why it is
harder to reproduce; different access pattern, slower read->less stress to
the IDE subsystem. (2.4 also does 30-35MB/s.)
(Mark Hahn suggested I should try to compare the blocks - not just md5sum
them - to see which bytes differ.)
I wrote a little program to try compare 512MB blocks. This turned out
difficult, since the only drive in addition to these two striped IDE drives
(that constitute md0) is an oldish 18GB Seagate scsi, where the rootfs
resides. The Seagate can only write max. 15MB/s, so I really can't stream
the 512MB block anywhere at full speed. And I do need to store them
somewhere to be able to reread and compare. The block can't be much smaller
than 512MB since I have 256MB of ram, and I absolutely do not want the OS to
pull the 2nd (comparison) read from the cache.
Right now I have the 2nd run going, but at these read speeds:
At reading block at 59392 MB...13.8 MB/s. Comparing...7.8 MB/s.
I don't expect the IDE system to be stressed enough yield errors. (Reading
includes read 512MB off md0 + write 512MB to sda, compare includes 512MB
read from md0 + 512MB read from sda)
Anyhow, at this point I'm not really that interested in what bytes differ.
HPT370+IBM-DPTA-373420 + linux (be it 2.2+IDE, 2.4 or 2.4+Hockin's hpt
patch) corrupt data, end of story.
I think the next stunt to pull is to plug the beasts on the VIA 686B IDE.
"Via 686B ide" doesn't have that trust-evoking sound either (which is why I
went the HPT370 path in the first place; I had had very good experinces with
HPT366 albeit albeit some consider that flaky as well). But, I guess I have
little to lose with 686B at this point...
-- v -
[email protected]