LinuxLists.cc - RAID superblock confusion

[permalink] [raw]

Subject: Re: RAID superblock confusion

Andreas Dilger writes:
> On Apr 10, 2002 09:33 -0600, Richard Gooch wrote:
> > Even though I'm using persistent superblockss, which is supposed to
> > allow one to move devices from one controller to another, I can't
> > use my RAID) set in this configuration. Looks like a bug.
> >
> > md0: former device scsi/host2/bus0/target1/lun0/part2 is unavailable, removing from array!
> > md: md0, array needs 6 disks, has 5, aborting.
>
> Note that this appears to be your real problem.

No. I tested all 6 partitions used in the RAID set. They are all
available.

> > Note the following line from the kernel logs above:
> > md: can not import scsi/host6/bus0/target0/lun0/part2, has active inodes!
> >
> > Well, that's no surprise, as this partition has /usr! And this
> > partition isn't even mentioned in the /etc/raidtab file. But I note
> > that it has the same device number in this (the broken) configuration
> > as /dev/sd/c0b0t1u0p2 has in the working configuration.
>
> That is a red herring, I think.

Then what *is* the problem?

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-10 19:39:42

by Andreas Dilger

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Apr 10, 2002 13:24 -0600, Richard Gooch wrote:
> Andreas Dilger writes:
> > On Apr 10, 2002 09:33 -0600, Richard Gooch wrote:
> > > Even though I'm using persistent superblockss, which is supposed to
> > > allow one to move devices from one controller to another, I can't
> > > use my RAID) set in this configuration. Looks like a bug.
> > >
> > > md0: former device scsi/host2/bus0/target1/lun0/part2 is unavailable,
> > > removing from array!
> > > md: md0, array needs 6 disks, has 5, aborting.
> >
> > Note that this appears to be your real problem.
>
> No. I tested all 6 partitions used in the RAID set. They are all
> available.

Well, MD seems to think it is unavailable... I would check the codepath
that generates this message and see why it is happening. Maybe it is a
timing issue or something, that MD autostart is starting before this
device is set up or something? I don't know.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

2002-04-10 20:37:57

[permalink] [raw]

Subject: Re: RAID superblock confusion

Andreas Dilger writes:
> On Apr 10, 2002 13:24 -0600, Richard Gooch wrote:
> > Andreas Dilger writes:
> > > On Apr 10, 2002 09:33 -0600, Richard Gooch wrote:
> > > > Even though I'm using persistent superblockss, which is supposed to
> > > > allow one to move devices from one controller to another, I can't
> > > > use my RAID) set in this configuration. Looks like a bug.
> > > >
> > > > md0: former device scsi/host2/bus0/target1/lun0/part2 is unavailable,
> > > > removing from array!
> > > > md: md0, array needs 6 disks, has 5, aborting.
> > >
> > > Note that this appears to be your real problem.
> >
> > No. I tested all 6 partitions used in the RAID set. They are all
> > available.
>
> Well, MD seems to think it is unavailable... I would check the
> codepath that generates this message and see why it is happening.
> Maybe it is a timing issue or something, that MD autostart is
> starting before this device is set up or something? I don't know.

The device is set up (i.e. SCSI host driver is loaded) long before I
do raidstart /dev/md/0

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-10 21:34:41

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wed, Apr 10, 2002 at 02:37:48PM -0600, Richard Gooch wrote:
> Andreas Dilger writes:
> > On Apr 10, 2002 13:24 -0600, Richard Gooch wrote:
> > > Andreas Dilger writes:
> > > > On Apr 10, 2002 09:33 -0600, Richard Gooch wrote:
> > > > > Even though I'm using persistent superblockss, which is supposed to
> > > > > allow one to move devices from one controller to another, I can't
> > > > > use my RAID) set in this configuration. Looks like a bug.
> > > > >
> > > > > md0: former device scsi/host2/bus0/target1/lun0/part2 is unavailable,
> > > > > removing from array!
> > > > > md: md0, array needs 6 disks, has 5, aborting.
> > > >
> > > > Note that this appears to be your real problem.
> > >
> > > No. I tested all 6 partitions used in the RAID set. They are all
> > > available.
> >
> > Well, MD seems to think it is unavailable... I would check the
> > codepath that generates this message and see why it is happening.
> > Maybe it is a timing issue or something, that MD autostart is
> > starting before this device is set up or something? I don't know.
>
> The device is set up (i.e. SCSI host driver is loaded) long before I
> do raidstart /dev/md/0

But kernel auto-detection doesn't depend on the raidstart command. If
things are setup correctly, you can remove that from your init scripts.

2002-04-10 21:40:31

[permalink] [raw]

Subject: Re: RAID superblock confusion

Mike Fedyk writes:
> On Wed, Apr 10, 2002 at 02:37:48PM -0600, Richard Gooch wrote:
> > Andreas Dilger writes:
> > > On Apr 10, 2002 13:24 -0600, Richard Gooch wrote:
> > > > Andreas Dilger writes:
> > > > > On Apr 10, 2002 09:33 -0600, Richard Gooch wrote:
> > > > > > Even though I'm using persistent superblockss, which is supposed to
> > > > > > allow one to move devices from one controller to another, I can't
> > > > > > use my RAID) set in this configuration. Looks like a bug.
> > > > > >
> > > > > > md0: former device scsi/host2/bus0/target1/lun0/part2 is unavailable,
> > > > > > removing from array!
> > > > > > md: md0, array needs 6 disks, has 5, aborting.
> > > > >
> > > > > Note that this appears to be your real problem.
> > > >
> > > > No. I tested all 6 partitions used in the RAID set. They are all
> > > > available.
> > >
> > > Well, MD seems to think it is unavailable... I would check the
> > > codepath that generates this message and see why it is happening.
> > > Maybe it is a timing issue or something, that MD autostart is
> > > starting before this device is set up or something? I don't know.
> >
> > The device is set up (i.e. SCSI host driver is loaded) long before I
> > do raidstart /dev/md/0
>
> But kernel auto-detection doesn't depend on the raidstart command. If
> things are setup correctly, you can remove that from your init scripts.

I'm not (explicitely) using auto-detection. When I insmod the raid0
module, there are no messages about finding devices. All I get is:
md: raid0 personality registered as nr 2

Only when I run raidstart do I get kernel messages about the devices.

In any case, I should be able to move my devices around (especially
if /etc/raidtab is still correct), whether or not autostart is
running. The behaviour I'm observing is a bug (I assume it's not a
mis-feature, since the raidstart man page tells me that moving devices
around should be safe).

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-10 22:07:26

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wed, Apr 10, 2002 at 03:39:09PM -0600, Richard Gooch wrote:
> Mike Fedyk writes:
> > On Wed, Apr 10, 2002 at 02:37:48PM -0600, Richard Gooch wrote:
> > >
> > > The device is set up (i.e. SCSI host driver is loaded) long before I
> > > do raidstart /dev/md/0
> >
> > But kernel auto-detection doesn't depend on the raidstart command. If
> > things are setup correctly, you can remove that from your init scripts.
>
> I'm not (explicitely) using auto-detection. When I insmod the raid0
> module, there are no messages about finding devices. All I get is:
> md: raid0 personality registered as nr 2
>
> Only when I run raidstart do I get kernel messages about the devices.
>
> In any case, I should be able to move my devices around (especially
> if /etc/raidtab is still correct), whether or not autostart is
> running. The behaviour I'm observing is a bug (I assume it's not a
> mis-feature, since the raidstart man page tells me that moving devices
> around should be safe).

Ehh, I ran into this a while ago. When you compile raid as modules it
doesn't use the raid superblocks for anything except for verification. I
took a quick glance at the source and the auto-detect code is ifdefed out if
you compiled as a module.

Ever since I have had raid compiled into my kernels.

Mike

2002-04-10 22:49:14

[permalink] [raw]

Subject: Re: RAID superblock confusion

Mike Fedyk writes:
> On Wed, Apr 10, 2002 at 03:39:09PM -0600, Richard Gooch wrote:
> > Mike Fedyk writes:
> > > On Wed, Apr 10, 2002 at 02:37:48PM -0600, Richard Gooch wrote:
> > > >
> > > > The device is set up (i.e. SCSI host driver is loaded) long before I
> > > > do raidstart /dev/md/0
> > >
> > > But kernel auto-detection doesn't depend on the raidstart command. If
> > > things are setup correctly, you can remove that from your init scripts.
> >
> > I'm not (explicitely) using auto-detection. When I insmod the raid0
> > module, there are no messages about finding devices. All I get is:
> > md: raid0 personality registered as nr 2
> >
> > Only when I run raidstart do I get kernel messages about the devices.
> >
> > In any case, I should be able to move my devices around (especially
> > if /etc/raidtab is still correct), whether or not autostart is
> > running. The behaviour I'm observing is a bug (I assume it's not a
> > mis-feature, since the raidstart man page tells me that moving devices
> > around should be safe).
>
> Ehh, I ran into this a while ago. When you compile raid as modules
> it doesn't use the raid superblocks for anything except for
> verification. I took a quick glance at the source and the
> auto-detect code is ifdefed out if you compiled as a module.

Exactly where is this? A scan with find and grep don't reveal this.

> Ever since I have had raid compiled into my kernels.

This is my relevant .config:
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID5=m
CONFIG_MD_MULTIPATH=m

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-10 23:34:33

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wed, Apr 10, 2002 at 04:49:06PM -0600, Richard Gooch wrote:
> Mike Fedyk writes:
> > On Wed, Apr 10, 2002 at 03:39:09PM -0600, Richard Gooch wrote:
> > > Mike Fedyk writes:
> > > > On Wed, Apr 10, 2002 at 02:37:48PM -0600, Richard Gooch wrote:
> > > > >
> > > > > The device is set up (i.e. SCSI host driver is loaded) long before I
> > > > > do raidstart /dev/md/0
> > > >
> > > > But kernel auto-detection doesn't depend on the raidstart command. If
> > > > things are setup correctly, you can remove that from your init scripts.
> > >
> > > I'm not (explicitely) using auto-detection. When I insmod the raid0
> > > module, there are no messages about finding devices. All I get is:
> > > md: raid0 personality registered as nr 2
> > >
> > > Only when I run raidstart do I get kernel messages about the devices.
> > >
> > > In any case, I should be able to move my devices around (especially
> > > if /etc/raidtab is still correct), whether or not autostart is
> > > running. The behaviour I'm observing is a bug (I assume it's not a
> > > mis-feature, since the raidstart man page tells me that moving devices
> > > around should be safe).
> >
> > Ehh, I ran into this a while ago. When you compile raid as modules
> > it doesn't use the raid superblocks for anything except for
> > verification. I took a quick glance at the source and the
> > auto-detect code is ifdefed out if you compiled as a module.
>
> Exactly where is this? A scan with find and grep don't reveal this.
>

drivers/md/md.c

in the ifndef MODULE sectioin.

> > Ever since I have had raid compiled into my kernels.
>
> This is my relevant .config:
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=m
> CONFIG_MD_RAID0=m
> CONFIG_MD_RAID1=m
> CONFIG_MD_RAID5=m
> CONFIG_MD_MULTIPATH=m
>

Set this to =y and you're set.

I'd like to see this working from modules though.

Mike

2002-04-11 01:35:16

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wednesday April 10, [email protected] wrote:
>
> The device is set up (i.e. SCSI host driver is loaded) long before I
> do raidstart /dev/md/0

raidstart simply does not and cannot work reliably when your device
numbers change around. It takes the first device listed in
/etc/raidtab and gives it to the kernel.
The kernel reads the superblock, finds some device numbers and tries
to attach those devices. If device number have changed, you loose.

autodetect is the other alternative. However, as has been mentioned,
it does not and cannot work with md as a module. This is because
devices can only be register for autodetection after md.o is loaded,
and autodetection is done at the time that md is loaded. So
autodetection can only work if the device driver and md are loaded at
simultaneously. i.e. they are compiled into the kernel.

I use and recommend mdadm.
http://www.cse.unsw.edu.au/~neilb/source/mdadm/
(I hope to release a 1.0 in the next week or so).

With mdadm you can assemble devices whenever you like based on uuid in
the superblock.

Assuming you use devfs :-)
Create /etc/mdadm.conf containing:

DEVICE /dev/scsi/*/*/*/*/part*

[[ you should be able to say

DEVICE /dev/discs/*/part*

but glob(3) gets confused by that :-(
]]

and run

mdadm -Es >> /etc/mdadm.conf

Then edit /etc/mdadm.conf removing bits that are wrong, and changing
bits that need changing. The mdadm.conf.5 to understand what should
be there.

You will need to:
get rid of the "devices=" parts
Make sure no extra arrays were found.

Once you have done this, then

mdadm -As

in your startup scripts will assemble your arrays.

Make sure you use mdadm-0.8.2. I found a little bug in 0.8.1 as I was
testing this example.

NeilBrown

2002-04-11 02:39:02

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Thu, Apr 11, 2002 at 11:38:19AM +1000, Neil Brown wrote:
> autodetect is the other alternative. However, as has been mentioned,
> it does not and cannot work with md as a module. This is because
> devices can only be register for autodetection after md.o is loaded,
> and autodetection is done at the time that md is loaded. So
> autodetection can only work if the device driver and md are loaded at
> simultaneously. i.e. they are compiled into the kernel.

Ahh, but if you use initrd you can even have the ide and scsi drivers as
modules.

What is needed is to make the disk modules depend on the raid modules (only
if the raid code is enabled of course) so that modprobe can load the raid
modules first.

Then you'd just need to make sure that if there are any block modules linked
into the kernel that raid is also linked into the kernle instead of a module.

Is there some reason why this wouldn't work (except for CML1
complications...)?

Kieth, will kbuild2.5 affect this in any way? Or is this entirely a CML2
thing?

Mike

2002-04-11 03:15:37

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wednesday April 10, [email protected] wrote:
> On Thu, Apr 11, 2002 at 11:38:19AM +1000, Neil Brown wrote:
> > autodetect is the other alternative. However, as has been mentioned,
> > it does not and cannot work with md as a module. This is because
> > devices can only be register for autodetection after md.o is loaded,
> > and autodetection is done at the time that md is loaded. So
> > autodetection can only work if the device driver and md are loaded at
> > simultaneously. i.e. they are compiled into the kernel.
>
> Ahh, but if you use initrd you can even have the ide and scsi drivers as
> modules.
>
> What is needed is to make the disk modules depend on the raid modules (only
> if the raid code is enabled of course) so that modprobe can load the raid
> modules first.
>
> Then you'd just need to make sure that if there are any block modules linked
> into the kernel that raid is also linked into the kernle instead of
> a module.

Woah... I think you are going off the deep end here. This sounds just
too complicated.

1/ If you wanted to do autodetect "right", you would make it look a
lot like partition detection (split md into two bits. A partition
detection personality that registers the component devices
somewhere, and the main raid module that gets autoloaded when you
try to actually access a raid device).
2/ Partition detection *should* be done in user-space. So should
autodetect. mdadm does that for you..

NeilBrown

2002-04-11 06:43:03

by Keith Owens

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wed, 10 Apr 2002 19:41:11 -0700,
Mike Fedyk <[email protected]> wrote:
>Then you'd just need to make sure that if there are any block modules linked
>into the kernel that raid is also linked into the kernle instead of a module.
>
>Is there some reason why this wouldn't work (except for CML1
>complications...)?
>
>Kieth, will kbuild2.5 affect this in any way? Or is this entirely a CML2
>thing?

AFAICT it is pure config. kbuild 2.5 does not care how .config is
built, it just requires a clean .config. I want a clean seperation
between configuring and building the kernel. In particular, Makefile
rules should not try to adjust for broken config entries, this is a
config only problem.

2002-04-11 08:39:00

by Helge Hafting

[permalink] [raw]

Subject: Re: RAID superblock confusion

Mike Fedyk wrote:
[...]
> Ahh, but if you use initrd you can even have the ide and scsi drivers as
> modules.
>
> What is needed is to make the disk modules depend on the raid modules (only
> if the raid code is enabled of course) so that modprobe can load the raid
> modules first.
>
> Then you'd just need to make sure that if there are any block modules linked
> into the kernel that raid is also linked into the kernle instead of a module.
>
> Is there some reason why this wouldn't work (except for CML1
> complications...)?

Having block devices depend on raid modules isn't the solution.
A user may have many block devices with raid on only a few
of them.

I think this sort of setup-specific dependencies belong in userspace,
i.e. /etc/conf.modules:

pre-install <block-driver-with-raid-partitions> insmod
<raid-modules-needed>

Helge Hafting

2002-04-11 10:07:47

[permalink] [raw]

Subject: Re: RAID superblock confusion

> > >
> > > Ehh, I ran into this a while ago. When you compile raid as modules
> > > it doesn't use the raid superblocks for anything except for
> > > verification. I took a quick glance at the source and the
> > > auto-detect code is ifdefed out if you compiled as a module.
> >
> > Exactly where is this? A scan with find and grep don't reveal this.
> >
>
> drivers/md/md.c
>
> in the ifndef MODULE sectioin.
>
> > > Ever since I have had raid compiled into my kernels.
> >
> > This is my relevant .config:
> > CONFIG_MD=y
> > CONFIG_BLK_DEV_MD=y
> > CONFIG_MD_LINEAR=m
> > CONFIG_MD_RAID0=m
> > CONFIG_MD_RAID1=m
> > CONFIG_MD_RAID5=m
> > CONFIG_MD_MULTIPATH=m
> >
>
> Set this to =y and you're set.
>
> I'd like to see this working from modules though.

NO, please. There are hundreds of scenarios where that could be dangerous.
Suppose you load the RAID
module when all partitions are mounted, and two partiton in mirror are
mount on different mount point (you can do this, raid module is not
loaded, and so...). And now you load the module and md device is
registered. That would not be really nice, also if it is ulikely that you
could damnage your system

2002-04-11 10:20:04

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Thu, 11 Apr 2002, Neil Brown wrote:

> On Wednesday April 10, [email protected] wrote:
> > On Thu, Apr 11, 2002 at 11:38:19AM +1000, Neil Brown wrote:
> > > autodetect is the other alternative. However, as has been mentioned,
> > > it does not and cannot work with md as a module. This is because
> > > devices can only be register for autodetection after md.o is loaded,
> > > and autodetection is done at the time that md is loaded. So
> > > autodetection can only work if the device driver and md are loaded at
> > > simultaneously. i.e. they are compiled into the kernel.
> >
> > Ahh, but if you use initrd you can even have the ide and scsi drivers as
> > modules.
> >
> > What is needed is to make the disk modules depend on the raid modules (only
> > if the raid code is enabled of course) so that modprobe can load the raid
> > modules first.
you are supposing that I load md modules and raid module together, mostly
during boot with initrd. In the reality I have some servers with more that
200 days of uptime, and I have to change external disks sometime. I do
usually have two external boxes, and something like 8/20 disks (two scsi
controllers), and different raid on different disks. You see, it is not
that easy.

Luigi

2002-04-11 20:16:56

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Thu, Apr 11, 2002 at 12:19:35PM +0200, Luigi Genoni wrote:
> > On Wednesday April 10, [email protected] wrote:
> > > On Thu, Apr 11, 2002 at 11:38:19AM +1000, Neil Brown wrote:
> > > > autodetect is the other alternative. However, as has been mentioned,
> > > > it does not and cannot work with md as a module. This is because
> > > > devices can only be register for autodetection after md.o is loaded,
> > > > and autodetection is done at the time that md is loaded. So
> > > > autodetection can only work if the device driver and md are loaded at
> > > > simultaneously. i.e. they are compiled into the kernel.
> > >
> > > Ahh, but if you use initrd you can even have the ide and scsi drivers as
> > > modules.
> > >
> > > What is needed is to make the disk modules depend on the raid modules (only
> > > if the raid code is enabled of course) so that modprobe can load the raid
> > > modules first.
> you are supposing that I load md modules and raid module together, mostly

md and raid is the same and not split yet as Niel proposed.

> during boot with initrd. In the reality I have some servers with more that
> 200 days of uptime, and I have to change external disks sometime. I do
> usually have two external boxes, and something like 8/20 disks (two scsi
> controllers), and different raid on different disks. You see, it is not
> that easy.

Currently, if you have raid compiled as modules the autodetection does not
work, and is disabled. The issue here is enabling autodetection from the
modules, wheather that be at boot time or not. If you use raid modules you
won't get the autodetection and the features that come with that.

When adding a new disk, you would have to add it to an array manually with
the raidtools2 anyway, so this is unchanged.

Mike

2002-04-13 19:26:23

[permalink] [raw]

Subject: Re: RAID superblock confusion

Neil Brown writes:
> On Wednesday April 10, [email protected] wrote:
> >
> > The device is set up (i.e. SCSI host driver is loaded) long before I
> > do raidstart /dev/md/0
>
> raidstart simply does not and cannot work reliably when your device
> numbers change around. It takes the first device listed in
> /etc/raidtab and gives it to the kernel.
> The kernel reads the superblock, finds some device numbers and tries
> to attach those devices. If device number have changed, you loose.

Sounds to me like the flaw is in the ioctl(2) interface, in that it
doesn't allow passing all the block devices in the RAID set. If it
allowed you to pass all the block devices, then it could check if all
the signatures on each block device match.

I tried the alternative of setting persistent-superblock=0 in
/etc/raidtab, but the stupid thing complained because it found a
superblock. Sigh.

If there was only a "do as I say, regardless" mode, I would be happy.
This programmer-knows-best attitude smacks of M$.

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-13 19:29:59

[permalink] [raw]

Subject: Re: RAID superblock confusion

Luigi Genoni writes:
>
> > > >
> > > > Ehh, I ran into this a while ago. When you compile raid as modules
> > > > it doesn't use the raid superblocks for anything except for
> > > > verification. I took a quick glance at the source and the
> > > > auto-detect code is ifdefed out if you compiled as a module.
> > >
> > > Exactly where is this? A scan with find and grep don't reveal this.
> > >
> >
> > drivers/md/md.c
> >
> > in the ifndef MODULE sectioin.
> >
> > > > Ever since I have had raid compiled into my kernels.
> > >
> > > This is my relevant .config:
> > > CONFIG_MD=y
> > > CONFIG_BLK_DEV_MD=y
> > > CONFIG_MD_LINEAR=m
> > > CONFIG_MD_RAID0=m
> > > CONFIG_MD_RAID1=m
> > > CONFIG_MD_RAID5=m
> > > CONFIG_MD_MULTIPATH=m
> > >
> >
> > Set this to =y and you're set.
> >
> > I'd like to see this working from modules though.
>
> NO, please. There are hundreds of scenarios where that could be
> dangerous. Suppose you load the RAID module when all partitions are
> mounted, and two partiton in mirror are mount on different mount
> point (you can do this, raid module is not loaded, and so...). And
> now you load the module and md device is registered. That would not
> be really nice, also if it is ulikely that you could damnage your
> system

The RAID code checks to see if there are busy inodes for each device
in a RAID set. So your hundreds of scenarios are not a problem.

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-13 23:53:23

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Sat, Apr 13, 2002 at 01:29:05PM -0600, Richard Gooch wrote:
> Luigi Genoni writes:
> >
> > > > >
> > > > > Ehh, I ran into this a while ago. When you compile raid as modules
> > > > > it doesn't use the raid superblocks for anything except for
> > > > > verification. I took a quick glance at the source and the
> > > > > auto-detect code is ifdefed out if you compiled as a module.
> > > >
> > > > Exactly where is this? A scan with find and grep don't reveal this.
> > > >
> > >
> > > drivers/md/md.c
> > >
> > > in the ifndef MODULE sectioin.
> > >
> > > > > Ever since I have had raid compiled into my kernels.
> > > >
> > > > This is my relevant .config:
> > > > CONFIG_MD=y
> > > > CONFIG_BLK_DEV_MD=y
> > > > CONFIG_MD_LINEAR=m
> > > > CONFIG_MD_RAID0=m
> > > > CONFIG_MD_RAID1=m
> > > > CONFIG_MD_RAID5=m
> > > > CONFIG_MD_MULTIPATH=m
> > > >
> > >
> > > Set this to =y and you're set.
> > >
> > > I'd like to see this working from modules though.
> >
> > NO, please. There are hundreds of scenarios where that could be
> > dangerous. Suppose you load the RAID module when all partitions are
> > mounted, and two partiton in mirror are mount on different mount
> > point (you can do this, raid module is not loaded, and so...). And
> > now you load the module and md device is registered. That would not
> > be really nice, also if it is ulikely that you could damnage your
> > system
>
> The RAID code checks to see if there are busy inodes for each device
> in a RAID set. So your hundreds of scenarios are not a problem.
>

I had a machine that had raid1 setup correctly but was accidentally
configured to root=/dev/hda1 (one member of the md0 raid1 set).

All was well until I noticed I wasn't rooting from md0, so reboot with new
root=/dev/md0 and now my filesystem is b0rked (maybe because hdc1 was the
primary mirror?).

Luckily I was still setting up that machine so I just reinstalled it.

This was with raid compiled into the kernel, so it's not a module checking
issue, and I consider it a user error. But maybe someone else thinks
different...

Just reporting in case someone is intereted...

Mike

2002-04-14 00:00:20

[permalink] [raw]

Subject: Re: RAID superblock confusion

Mike Fedyk writes:
> On Sat, Apr 13, 2002 at 01:29:05PM -0600, Richard Gooch wrote:
> > Luigi Genoni writes:
> > >
> > > > > >
> > > > > > Ehh, I ran into this a while ago. When you compile raid as modules
> > > > > > it doesn't use the raid superblocks for anything except for
> > > > > > verification. I took a quick glance at the source and the
> > > > > > auto-detect code is ifdefed out if you compiled as a module.
> > > > >
> > > > > Exactly where is this? A scan with find and grep don't reveal this.
> > > > >
> > > >
> > > > drivers/md/md.c
> > > >
> > > > in the ifndef MODULE sectioin.
> > > >
> > > > > > Ever since I have had raid compiled into my kernels.
> > > > >
> > > > > This is my relevant .config:
> > > > > CONFIG_MD=y
> > > > > CONFIG_BLK_DEV_MD=y
> > > > > CONFIG_MD_LINEAR=m
> > > > > CONFIG_MD_RAID0=m
> > > > > CONFIG_MD_RAID1=m
> > > > > CONFIG_MD_RAID5=m
> > > > > CONFIG_MD_MULTIPATH=m
> > > > >
> > > >
> > > > Set this to =y and you're set.
> > > >
> > > > I'd like to see this working from modules though.
> > >
> > > NO, please. There are hundreds of scenarios where that could be
> > > dangerous. Suppose you load the RAID module when all partitions are
> > > mounted, and two partiton in mirror are mount on different mount
> > > point (you can do this, raid module is not loaded, and so...). And
> > > now you load the module and md device is registered. That would not
> > > be really nice, also if it is ulikely that you could damnage your
> > > system
> >
> > The RAID code checks to see if there are busy inodes for each device
> > in a RAID set. So your hundreds of scenarios are not a problem.
> >
>
> I had a machine that had raid1 setup correctly but was accidentally
> configured to root=/dev/hda1 (one member of the md0 raid1 set).
>
> All was well until I noticed I wasn't rooting from md0, so reboot with new
> root=/dev/md0 and now my filesystem is b0rked (maybe because hdc1 was the
> primary mirror?).
>
> Luckily I was still setting up that machine so I just reinstalled it.
>
> This was with raid compiled into the kernel, so it's not a module checking
> issue, and I consider it a user error. But maybe someone else thinks
> different...

Yep, user error. Just like if you dd if=/dev/zero of=/dev/hda2 but
meant to write to /dev/hda1 instead, and /dev/hda2 has your OS while
/dev/hda1 had M$ which you wanted to erase and re-install. Not much to
be done about that. One learns best by fucking up :-)

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-14 00:15:24

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Sat, Apr 13, 2002 at 06:00:13PM -0600, Richard Gooch wrote:
> Mike Fedyk writes:
> > On Sat, Apr 13, 2002 at 01:29:05PM -0600, Richard Gooch wrote:
> > > Luigi Genoni writes:
> > > >
> > > > > > >
> > > > > > > Ehh, I ran into this a while ago. When you compile raid as modules
> > > > > > > it doesn't use the raid superblocks for anything except for
> > > > > > > verification. I took a quick glance at the source and the
> > > > > > > auto-detect code is ifdefed out if you compiled as a module.
> > > > > >
> > > > > > Exactly where is this? A scan with find and grep don't reveal this.
> > > > > >
> > > > >
> > > > > drivers/md/md.c
> > > > >
> > > > > in the ifndef MODULE sectioin.
> > > > >
> > > > > > > Ever since I have had raid compiled into my kernels.
> > > > > >
> > > > > > This is my relevant .config:
> > > > > > CONFIG_MD=y
> > > > > > CONFIG_BLK_DEV_MD=y
> > > > > > CONFIG_MD_LINEAR=m
> > > > > > CONFIG_MD_RAID0=m
> > > > > > CONFIG_MD_RAID1=m
> > > > > > CONFIG_MD_RAID5=m
> > > > > > CONFIG_MD_MULTIPATH=m
> > > > > >
> > > > >
> > > > > Set this to =y and you're set.
> > > > >
> > > > > I'd like to see this working from modules though.
> > > >
> > > > NO, please. There are hundreds of scenarios where that could be
> > > > dangerous. Suppose you load the RAID module when all partitions are
> > > > mounted, and two partiton in mirror are mount on different mount
> > > > point (you can do this, raid module is not loaded, and so...). And
> > > > now you load the module and md device is registered. That would not
> > > > be really nice, also if it is ulikely that you could damnage your
> > > > system
> > >
> > > The RAID code checks to see if there are busy inodes for each device
> > > in a RAID set. So your hundreds of scenarios are not a problem.
> > >
> >
> > I had a machine that had raid1 setup correctly but was accidentally
> > configured to root=/dev/hda1 (one member of the md0 raid1 set).
> >
> > All was well until I noticed I wasn't rooting from md0, so reboot with new
> > root=/dev/md0 and now my filesystem is b0rked (maybe because hdc1 was the
> > primary mirror?).
> >
> > Luckily I was still setting up that machine so I just reinstalled it.
> >
> > This was with raid compiled into the kernel, so it's not a module checking
> > issue, and I consider it a user error. But maybe someone else thinks
> > different...
>
> Yep, user error. Just like if you dd if=/dev/zero of=/dev/hda2 but
> meant to write to /dev/hda1 instead, and /dev/hda2 has your OS while
> /dev/hda1 had M$ which you wanted to erase and re-install. Not much to
> be done about that. One learns best by fucking up :-)
>

One can also argue that raid (sh|c)ould lock the devices that it has been
started on also...

But it is the unix way to give the root user lots of rope to hang
him/herself, and what keeps the root user from trying to erase the entire
drive (hda instead of hda1) instead of just the locked partition?

2002-04-18 01:50:46

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Saturday April 13, [email protected] wrote:
> Neil Brown writes:
> > On Wednesday April 10, [email protected] wrote:
> > >
> > > The device is set up (i.e. SCSI host driver is loaded) long before I
> > > do raidstart /dev/md/0
> >
> > raidstart simply does not and cannot work reliably when your device
> > numbers change around. It takes the first device listed in
> > /etc/raidtab and gives it to the kernel.
> > The kernel reads the superblock, finds some device numbers and tries
> > to attach those devices. If device number have changed, you loose.
>
> Sounds to me like the flaw is in the ioctl(2) interface, in that it
> doesn't allow passing all the block devices in the RAID set. If it
> allowed you to pass all the block devices, then it could check if all
> the signatures on each block device match.

Exactly. The flaw is in the ioctl interface. 2.4 comes with an
improved ioctl interface which allows you to tell the kernel exactly
what you want it to do. mdadm used this interface.

>
> I tried the alternative of setting persistent-superblock=0 in
> /etc/raidtab, but the stupid thing complained because it found a
> superblock. Sigh.
>
> If there was only a "do as I say, regardless" mode, I would be happy.
> This programmer-knows-best attitude smacks of M$.

mdadm will do as you say, reguardless - if you ask it to. Have you
tried mdadm?
http://www.cse.unsw.edu.au/~neilb/source/mdadm/

NeilBrown

2002-04-18 02:13:45

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Thu, Apr 18, 2002 at 11:54:13AM +1000, Neil Brown wrote:
> On Saturday April 13, [email protected] wrote:
> > If there was only a "do as I say, regardless" mode, I would be happy.
> > This programmer-knows-best attitude smacks of M$.
>
> mdadm will do as you say, reguardless - if you ask it to. Have you
> tried mdadm?
> http://www.cse.unsw.edu.au/~neilb/source/mdadm/

Niel, do you plan to merge mdadm into the raidtools package? It sounds like
it belongs there.

2002-04-18 02:20:16

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wednesday April 17, [email protected] wrote:
> On Thu, Apr 18, 2002 at 11:54:13AM +1000, Neil Brown wrote:
> > On Saturday April 13, [email protected] wrote:
> > > If there was only a "do as I say, regardless" mode, I would be happy.
> > > This programmer-knows-best attitude smacks of M$.
> >
> > mdadm will do as you say, reguardless - if you ask it to. Have you
> > tried mdadm?
> > http://www.cse.unsw.edu.au/~neilb/source/mdadm/
>
> Niel, do you plan to merge mdadm into the raidtools package? It sounds like
> it belongs there.

No.
If distributions want to distribute mdadm together with the stuff from
raidtools, then that is up to them.
But from a development perspective, I don't see any value in making a
single source distribution.

NeilBrown

2002-04-18 02:57:10

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Thu, Apr 18, 2002 at 12:23:38PM +1000, Neil Brown wrote:
> On Wednesday April 17, [email protected] wrote:
> > On Thu, Apr 18, 2002 at 11:54:13AM +1000, Neil Brown wrote:
> > > On Saturday April 13, [email protected] wrote:
> > > > If there was only a "do as I say, regardless" mode, I would be happy.
> > > > This programmer-knows-best attitude smacks of M$.
> > >
> > > mdadm will do as you say, reguardless - if you ask it to. Have you
> > > tried mdadm?
> > > http://www.cse.unsw.edu.au/~neilb/source/mdadm/
> >
> > Niel, do you plan to merge mdadm into the raidtools package? It sounds like
> > it belongs there.
>
> No.
> If distributions want to distribute mdadm together with the stuff from
> raidtools, then that is up to them.
> But from a development perspective, I don't see any value in making a
> single source distribution.

Why's that? Don't they compliment each other, or is mdadm a replacement?
If it's not a replacement, combining it with raidtools2 would probably get
it into distributions faster and people have already been taught with
countless how-tos that you need the raidtools package...

Hmm, checking now debian has already packaged mdadm for debian 3.0 (woody),
so raidtools just may become obsoleted so my argument is probably moot...

ehh, what's one more message on lkml anyway? ;)

Mike

2002-04-18 03:02:21

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Wednesday April 17, [email protected] wrote:
> Why's that? Don't they compliment each other, or is mdadm a replacement?

You should read the man page....

>
> ehh, what's one more message on lkml anyway? ;)

Probably adds up to a person hour or so.

NeilBrown

2002-04-18 14:49:38

[permalink] [raw]

Subject: Re: RAID superblock confusion

Neil Brown writes:
> On Wednesday April 17, [email protected] wrote:
> > On Thu, Apr 18, 2002 at 11:54:13AM +1000, Neil Brown wrote:
> > > On Saturday April 13, [email protected] wrote:
> > > > If there was only a "do as I say, regardless" mode, I would be happy.
> > > > This programmer-knows-best attitude smacks of M$.
> > >
> > > mdadm will do as you say, reguardless - if you ask it to. Have you
> > > tried mdadm?
> > > http://www.cse.unsw.edu.au/~neilb/source/mdadm/
> >
> > Niel, do you plan to merge mdadm into the raidtools package? It sounds like
> > it belongs there.
>
> No.
> If distributions want to distribute mdadm together with the stuff from
> raidtools, then that is up to them.
> But from a development perspective, I don't see any value in making a
> single source distribution.

But from a user perspective, it's a pain the in ass to have to
download yet another package to do what raidtools do, just do it
properly. And having multiple packages doing nearly the same thing is
really confusing. Users won't know which one to pick.

<rant>
Every wanker out there wants to write their own cool tool, and since
sound and graphics are both l33t, why not do both at once? The result
is that we have (to name one example) a plethora of audio mixer
control tools.
Since I'm not in the "in crowd", and I just want a tool that lets me
frob the mixer, I wasted lots of time downloading many different
tools, reading the README's and compiling ones that didn't depend on
some bloated library (glibc, KDE or Gnome). Then waste more time
finding out which ones actually worked properly.
If resources were pooled, rather than all this ego stroking, we might
have something that (most importantly) works, is lightweight and is
usable for everyone, not just the users of the One True Desktop[R].
</rant>

Please let's just have one set of MD utilities.

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-19 13:46:46

[permalink] [raw]

Subject: Re: RAID superblock confusion

On Thu, 18 Apr 2002, Richard Gooch wrote:

> Since I'm not in the "in crowd", and I just want a tool that lets me
> frob the mixer, I wasted lots of time downloading many different
> tools, reading the README's and compiling ones that didn't depend on
> some bloated library (glibc, KDE or Gnome). Then waste more time
> finding out which ones actually worked properly.

I think you would admitt that it is quite difficoult di find a C source
code that does not depend on glibc ;).

Luigi

2002-04-19 13:49:59

[permalink] [raw]

Subject: Re: RAID superblock confusion

Luigi Genoni writes:
>
>
> On Thu, 18 Apr 2002, Richard Gooch wrote:
>
> > Since I'm not in the "in crowd", and I just want a tool that lets me
> > frob the mixer, I wasted lots of time downloading many different
> > tools, reading the README's and compiling ones that didn't depend on
> > some bloated library (glibc, KDE or Gnome). Then waste more time
> > finding out which ones actually worked properly.
>
> I think you would admitt that it is quite difficoult di find a C source
> code that does not depend on glibc ;).

glibc != only libc available.

Regards,

Richard....
Permanent: [email protected]
Current: [email protected]

2002-04-20 00:51:01