I have another report against rc1 also, but this is with a different line
number and under different circumstances, but with the same hardware.
Details in dmesg file.
On Tuesday August 19, [email protected] wrote:
> I have another report against rc1 also, but this is with a different line
> number and under different circumstances, but with the same hardware.
>
> Details in dmesg file.
.....
> Aug 15 16:44:22 srv-lr2600 kernel: RAID5 conf printout:
> Aug 15 16:44:22 srv-lr2600 kernel: --- rd:3 wd:2 fd:1
> Aug 15 16:44:22 srv-lr2600 kernel: disk 0, s:0, o:1, n:0 rd:0 us:1 dev:hde3
> Aug 15 16:44:22 srv-lr2600 kernel: disk 1, s:0, o:1, n:1 rd:1 us:1 dev:hdg3
> Aug 15 16:44:22 srv-lr2600 kernel: disk 2, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
> Aug 15 16:44:22 srv-lr2600 kernel: md: bug in file raid5.c, line 1909
>
> And why did I get this bug?
>
> Aug 15 16:44:22 srv-lr2600 kernel:
> Aug 15 16:44:22 srv-lr2600 kernel: md:^I**********************************
> Aug 15 16:44:22 srv-lr2600 kernel: md:^I* <COMPLETE RAID STATE PRINTOUT> *
> Aug 15 16:44:22 srv-lr2600 kernel: md:^I**********************************
> Aug 15 16:44:22 srv-lr2600 kernel: md0: <hda3><hdg3><hde3> array superblock:
> Aug 15 16:44:22 srv-lr2600 kernel: md: SB: (V:0.90.0) ID:<dea08cef.28d34b00.79cd55bc.46bdbe06> CT:3f34718d
> Aug 15 16:44:22 srv-lr2600 kernel: md: L5 S159694016 ND:3 RD:3 md0 LO:0 CS:65536
> Aug 15 16:44:23 srv-lr2600 kernel: md: UT:3f3d602e ST:0 AD:2 WD:3 FD:0 SD:1 CSUM:f9253789 E:0001b800
> Aug 15 16:44:23 srv-lr2600 kernel: D 0: DISK<N:0,hde3(33,3),R:0,S:6>
> Aug 15 16:44:23 srv-lr2600 kernel: D 1: DISK<N:1,hdg3(34,3),R:1,S:6>
> Aug 15 16:44:23 srv-lr2600 kernel: D 2: DISK<N:2,[dev 00:00](0,0),R:2,S:8>
> Aug 15 16:44:23 srv-lr2600 kernel: D 3: DISK<N:3,hda3(3,3),R:3,S:0>
Because descriptor 2 (D 2:) in the superblock has state
MD_DISK_REMOVED (S:8) rather and doesn't have the MD_DISK_FAULTY but
set (S:9 or S:1).
As far as I can see, the 2.4 code would never set just MD_DISK_REMOVED
(though it really should cope with it). It is possible that the 2.6
code does. Has this array had 2.6 running on it? Does it have any
interesting history?
NeilBrown
On Fri, Aug 22, 2003 at 03:22:30PM +1000, Neil Brown wrote:
> On Tuesday August 19, [email protected] wrote:
> > Aug 15 16:44:22 srv-lr2600 kernel: md: bug in file raid5.c, line 1909
> >
> > And why did I get this bug?
> >
> > Aug 15 16:44:22 srv-lr2600 kernel:
> > Aug 15 16:44:22 srv-lr2600 kernel: md:^I**********************************
> > Aug 15 16:44:22 srv-lr2600 kernel: md:^I* <COMPLETE RAID STATE PRINTOUT> *
> > Aug 15 16:44:22 srv-lr2600 kernel: md:^I**********************************
> > Aug 15 16:44:22 srv-lr2600 kernel: md0: <hda3><hdg3><hde3> array superblock:
> > Aug 15 16:44:22 srv-lr2600 kernel: md: SB: (V:0.90.0) ID:<dea08cef.28d34b00.79cd55bc.46bdbe06> CT:3f34718d
> > Aug 15 16:44:22 srv-lr2600 kernel: md: L5 S159694016 ND:3 RD:3 md0 LO:0 CS:65536
> > Aug 15 16:44:23 srv-lr2600 kernel: md: UT:3f3d602e ST:0 AD:2 WD:3 FD:0 SD:1 CSUM:f9253789 E:0001b800
> > Aug 15 16:44:23 srv-lr2600 kernel: D 0: DISK<N:0,hde3(33,3),R:0,S:6>
> > Aug 15 16:44:23 srv-lr2600 kernel: D 1: DISK<N:1,hdg3(34,3),R:1,S:6>
> > Aug 15 16:44:23 srv-lr2600 kernel: D 2: DISK<N:2,[dev 00:00](0,0),R:2,S:8>
> > Aug 15 16:44:23 srv-lr2600 kernel: D 3: DISK<N:3,hda3(3,3),R:3,S:0>
>
>
> Because descriptor 2 (D 2:) in the superblock has state
> MD_DISK_REMOVED (S:8) rather and doesn't have the MD_DISK_FAULTY but
> set (S:9 or S:1).
> As far as I can see, the 2.4 code would never set just MD_DISK_REMOVED
> (though it really should cope with it). It is possible that the 2.6
> code does. Has this array had 2.6 running on it? Does it have any
> interesting history?
Yes, it was running 2.6-test2-mm2 or so (don't remember exactly, I can check
though if needed) previously, but I didn't notice any bug messages from
there, and seeing that it was 2.4 I was surprised to see bug messages from md.
This is an IDE raid of three 160GB Maxtor hard drives. Unfortunately, a rw
badblocks run didn't show the problems that showed up while trying to setup
the production system, but smartctl did, and so did the kernel while 100
sectors were being moved around by the smart firmware in the drive. Anyway,
I'm getting a new drive, but after a few badblocks runs, and running a
couple raid resyncs (not my intention, but raid got funny after 2.6 was run
on the machine.)
Do you have any patches for 2.6 md? Right now this system is still in
testing, and I'd like to help get this code path tested, and fixed.
Mike
On Friday August 22, [email protected] wrote:
> > As far as I can see, the 2.4 code would never set just MD_DISK_REMOVED
> > (though it really should cope with it). It is possible that the 2.6
> > code does. Has this array had 2.6 running on it? Does it have any
> > interesting history?
>
> Yes, it was running 2.6-test2-mm2 or so (don't remember exactly, I can check
> though if needed) previously, but I didn't notice any bug messages from
> there, and seeing that it was 2.4 I was surprised to see bug
> messages from md.
The 2.4 code is very fragile. It can easily bug if the superblock
looks wrong in some obscure way.
>
> Do you have any patches for 2.6 md? Right now this system is still in
> testing, and I'd like to help get this code path tested, and fixed.
The following should fix it.
With this applied to 2.6, you can simply
start the array under 2.6
stop the array
reboot into 2.4
and it should be fine again.
NeilBrown
==================================================
Fix md superblock incompatabilities with 2.4 kernels.
2.4 kernels are very fussy about some values in the superblock, and
2.6 got them wrong. This fixes it.
----------- Diffstat output ------------
./drivers/md/md.c | 13 ++++++-------
1 files changed, 6 insertions(+), 7 deletions(-)
diff ./drivers/md/md.c~current~ ./drivers/md/md.c
--- ./drivers/md/md.c~current~ 2003-08-24 08:07:18.000000000 +1000
+++ ./drivers/md/md.c 2003-08-26 09:11:39.000000000 +1000
@@ -638,14 +638,13 @@ static void super_90_sync(mddev_t *mddev
/* make rdev->sb match mddev data..
*
* 1/ zero out disks
- * 2/ Add info for each disk, keeping track of highest desc_nr
- * 3/ any empty disks < highest become removed
+ * 2/ Add info for each disk, keeping track of highest desc_nr (next_spare);
+ * 3/ any empty disks < next_spare become removed
*
* disks[0] gets initialised to REMOVED because
* we cannot be sure from other fields if it has
* been initialised or not.
*/
- int highest = 0;
int i;
int active=0, working=0,failed=0,spare=0,nr_disks=0;
@@ -716,17 +715,17 @@ static void super_90_sync(mddev_t *mddev
spare++;
working++;
}
- if (rdev2->desc_nr > highest)
- highest = rdev2->desc_nr;
}
- /* now set the "removed" bit on any non-trailing holes */
- for (i=0; i<highest; i++) {
+ /* now set the "removed" and "faulty" bits on any missing devices */
+ for (i=0 ; i < mddev->raid_disks ; i++) {
mdp_disk_t *d = &sb->disks[i];
if (d->state == 0 && d->number == 0) {
d->number = i;
d->raid_disk = i;
d->state = (1<<MD_DISK_REMOVED);
+ d->state |= (1<<MD_DISK_FAULTY);
+ failed++;
}
}
sb->nr_disks = nr_disks;
On Tue, Aug 26, 2003 at 09:20:05AM +1000, Neil Brown wrote:
> On Friday August 22, [email protected] wrote:
> > > As far as I can see, the 2.4 code would never set just MD_DISK_REMOVED
> > > (though it really should cope with it). It is possible that the 2.6
> > > code does. Has this array had 2.6 running on it? Does it have any
> > > interesting history?
> >
> > Yes, it was running 2.6-test2-mm2 or so (don't remember exactly, I can check
> > though if needed) previously, but I didn't notice any bug messages from
> > there, and seeing that it was 2.4 I was surprised to see bug
> > messages from md.
>
> The 2.4 code is very fragile. It can easily bug if the superblock
> looks wrong in some obscure way.
>
> >
> > Do you have any patches for 2.6 md? Right now this system is still in
> > testing, and I'd like to help get this code path tested, and fixed.
>
> The following should fix it.
> With this applied to 2.6, you can simply
> start the array under 2.6
> stop the array
> reboot into 2.4
> and it should be fine again.
>
> NeilBrown
>
> ==================================================
> Fix md superblock incompatabilities with 2.4 kernels.
>
> 2.4 kernels are very fussy about some values in the superblock, and
> 2.6 got them wrong. This fixes it.
>
Yes, I can confirm this fixes it for me. I applied it to
2.6.0-test4-mm3-1, without any troubles.
I tried rebuilding the array with 2.4 (zeroing the superblock in mdadm and
everything), but that didn't fix it, so I rebuilt with 2.6 and the below
patch, and do far so good. I'll be beating on this machine some more...
Mike
>
> ----------- Diffstat output ------------
> ./drivers/md/md.c | 13 ++++++-------
> 1 files changed, 6 insertions(+), 7 deletions(-)
>
> diff ./drivers/md/md.c~current~ ./drivers/md/md.c
> --- ./drivers/md/md.c~current~ 2003-08-24 08:07:18.000000000 +1000
> +++ ./drivers/md/md.c 2003-08-26 09:11:39.000000000 +1000
> @@ -638,14 +638,13 @@ static void super_90_sync(mddev_t *mddev
> /* make rdev->sb match mddev data..
> *
> * 1/ zero out disks
> - * 2/ Add info for each disk, keeping track of highest desc_nr
> - * 3/ any empty disks < highest become removed
> + * 2/ Add info for each disk, keeping track of highest desc_nr (next_spare);
> + * 3/ any empty disks < next_spare become removed
> *
> * disks[0] gets initialised to REMOVED because
> * we cannot be sure from other fields if it has
> * been initialised or not.
> */
> - int highest = 0;
> int i;
> int active=0, working=0,failed=0,spare=0,nr_disks=0;
>
> @@ -716,17 +715,17 @@ static void super_90_sync(mddev_t *mddev
> spare++;
> working++;
> }
> - if (rdev2->desc_nr > highest)
> - highest = rdev2->desc_nr;
> }
>
> - /* now set the "removed" bit on any non-trailing holes */
> - for (i=0; i<highest; i++) {
> + /* now set the "removed" and "faulty" bits on any missing devices */
> + for (i=0 ; i < mddev->raid_disks ; i++) {
> mdp_disk_t *d = &sb->disks[i];
> if (d->state == 0 && d->number == 0) {
> d->number = i;
> d->raid_disk = i;
> d->state = (1<<MD_DISK_REMOVED);
> + d->state |= (1<<MD_DISK_FAULTY);
> + failed++;
> }
> }
> sb->nr_disks = nr_disks;
>