I am hoping someone here will help me. Was reading this site...
https://raid.wiki.kernel.org/index.php/Linux_Raid
and it said to email this list if you've tried everything other than
mdadm --create.
I am running Ubuntu 16.04. Machine name is fred. I used webmin to create
a 4 disk RAID10 array yesterday. I moved all my data onto the array.
Today, I had to reboot my PC. The resync was still not done, but I read
online that it's OK to boot during resync. After boot, my array was
gone. I checked syslog, and it just has this line:
DeviceDisappeared event detected on md device /dev/md0
I did not partition my disks before building the array. So I believe the
array consisted of /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf.
Here's some info...
stephen@fred> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 117.4G 0 disk
├─sda1 8:1 0 109.7G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 7.7G 0 part [SWAP]
sdb 8:16 0 465.8G 0 disk
└─sdb1 8:17 0 465.8G 0 part
sdc 8:32 0 3.7T 0 disk
sdd 8:48 0 3.7T 0 disk
sde 8:64 0 3.7T 0 disk
sdf 8:80 0 3.7T 0 disk
stephen@fred> sudo mdadm --examine /dev/sdc
[sudo] password for stephen:
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
stephen@fred>
stephen@fred> sudo mdadm --examine /dev/sdc1
mdadm: cannot open /dev/sdc1: No such file or directory
stephen@fred>
stephen@fred> sudo mdadm --examine /dev/sdd
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
stephen@fred>
stephen@fred> sudo mdadm --examine /dev/sde
/dev/sde:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
stephen@fred>
stephen@fred> sudo mdadm --examine /dev/sdf
/dev/sdf:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
stephen@fred> sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd
/dev/sde /dev/sdf
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: /dev/sdc has no superblock - assembly aborted
Thank you for any help you can provide.
On Thu, Mar 23, 2017 at 05:49:05PM +0100, [email protected] wrote:
> I am hoping someone here will help me. Was reading this site...
>
> https://raid.wiki.kernel.org/index.php/Linux_Raid
>
> and it said to email this list if you've tried everything other than mdadm
> --create.
>
>
> I am running Ubuntu 16.04. Machine name is fred. I used webmin to create a 4
> disk RAID10 array yesterday. I moved all my data onto the array.
>
> Today, I had to reboot my PC. The resync was still not done, but I read
> online that it's OK to boot during resync. After boot, my array was gone. I
> checked syslog, and it just has this line:
>
> DeviceDisappeared event detected on md device /dev/md0
>
> I did not partition my disks before building the array. So I believe the
> array consisted of /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf.
>
> Here's some info...
>
> stephen@fred> lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 117.4G 0 disk
> ├─sda1 8:1 0 109.7G 0 part /
> ├─sda2 8:2 0 1K 0 part
> └─sda5 8:5 0 7.7G 0 part [SWAP]
> sdb 8:16 0 465.8G 0 disk
> └─sdb1 8:17 0 465.8G 0 part
> sdc 8:32 0 3.7T 0 disk
> sdd 8:48 0 3.7T 0 disk
> sde 8:64 0 3.7T 0 disk
> sdf 8:80 0 3.7T 0 disk
>
> stephen@fred> sudo mdadm --examine /dev/sdc
> [sudo] password for stephen:
> /dev/sdc:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> stephen@fred>
> stephen@fred> sudo mdadm --examine /dev/sdc1
> mdadm: cannot open /dev/sdc1: No such file or directory
> stephen@fred>
> stephen@fred> sudo mdadm --examine /dev/sdd
> /dev/sdd:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> stephen@fred>
> stephen@fred> sudo mdadm --examine /dev/sde
> /dev/sde:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> stephen@fred>
> stephen@fred> sudo mdadm --examine /dev/sdf
> /dev/sdf:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
>
> stephen@fred> sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd
> /dev/sde /dev/sdf
> mdadm: Cannot assemble mbr metadata on /dev/sdc
> mdadm: /dev/sdc has no superblock - assembly aborted
>
> Thank you for any help you can provide.
Did your disks have partitions previously? That output looks a lot like
the protective MBR partition table for a disk with GPT partitions.
Could that still existing in sector 0 be confusing mdadm?
I have never personally done any md raid without partitions. To me they
just make more sense.
One way to test could be to save a copy of sector 0, then overwrite sector
0 with zeros and then run mdadm --examine again to see if that makes a
difference. You can always put back the saved copy of sector 0 that way.
My understanding is that the default is to put the raid superblock at
offset 4k, so it would not overwrite an existing MBR partition table.
If it also happens due to rounding that the end of the disk isn't
overwritten (or even just because that part of the filesystem wasn't
written to yet), then the backup GPT from before would still be intact,
and could perhaps cause even more confussion later if gdisk or similar
is pointed at the disk. Really want to be sure there is no trace left
of the partition table before using it raw for md raid.
Any chance the system saved an mdadm.conf file of your setup?
--
Len Sorensen
Thank you very much or your reply.
I naively thought that starting without partitions would be the best
starting point, given 3 of the disks had been in a RAID5 array
previously (possibly with partitions, not sure), but that looks like
a bad choice, based on some other things I've googled. Lesson learned.
I have an mdadm.conf file, but it could be a remnant of my previous
array. I've already edited it trying to get things to work, so I'm
not sure if it was updated when I created the new array or not.
I see various people online have had success in my situation using
madadm --create /dev/md0 --assume-clean --verbose --level=10 \
--raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf
Some people used --assume-clean, and some didn't. Given my array wasn't
done with its resync, maybe I should leave that out.
If that would work, I guess then I need to get the data off the array,
delete it, and recreate it with disk partitions, or risk this happening
again at the next reboot, for whatever reason.
Anyone think it's a bad idea to try mdadm --create at this point?
Sorry, I'm not sure how to write 0's to sector 0...
Thank you.
On 3/23/2017 18:37, Lennart Sorensen wrote:
> On Thu, Mar 23, 2017 at 05:49:05PM +0100, [email protected] wrote:
>> I am hoping someone here will help me. Was reading this site...
>>
>> https://raid.wiki.kernel.org/index.php/Linux_Raid
>>
>> and it said to email this list if you've tried everything other than mdadm
>> --create.
>>
>>
>> I am running Ubuntu 16.04. Machine name is fred. I used webmin to create a 4
>> disk RAID10 array yesterday. I moved all my data onto the array.
>>
>> Today, I had to reboot my PC. The resync was still not done, but I read
>> online that it's OK to boot during resync. After boot, my array was gone. I
>> checked syslog, and it just has this line:
>>
>> DeviceDisappeared event detected on md device /dev/md0
>>
>> I did not partition my disks before building the array. So I believe the
>> array consisted of /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf.
>>
>> Here's some info...
>>
>> stephen@fred> lsblk
>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>> sda 8:0 0 117.4G 0 disk
>> ├─sda1 8:1 0 109.7G 0 part /
>> ├─sda2 8:2 0 1K 0 part
>> └─sda5 8:5 0 7.7G 0 part [SWAP]
>> sdb 8:16 0 465.8G 0 disk
>> └─sdb1 8:17 0 465.8G 0 part
>> sdc 8:32 0 3.7T 0 disk
>> sdd 8:48 0 3.7T 0 disk
>> sde 8:64 0 3.7T 0 disk
>> sdf 8:80 0 3.7T 0 disk
>>
>> stephen@fred> sudo mdadm --examine /dev/sdc
>> [sudo] password for stephen:
>> /dev/sdc:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> stephen@fred>
>> stephen@fred> sudo mdadm --examine /dev/sdc1
>> mdadm: cannot open /dev/sdc1: No such file or directory
>> stephen@fred>
>> stephen@fred> sudo mdadm --examine /dev/sdd
>> /dev/sdd:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> stephen@fred>
>> stephen@fred> sudo mdadm --examine /dev/sde
>> /dev/sde:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>> stephen@fred>
>> stephen@fred> sudo mdadm --examine /dev/sdf
>> /dev/sdf:
>> MBR Magic : aa55
>> Partition[0] : 4294967295 sectors at 1 (type ee)
>>
>> stephen@fred> sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd
>> /dev/sde /dev/sdf
>> mdadm: Cannot assemble mbr metadata on /dev/sdc
>> mdadm: /dev/sdc has no superblock - assembly aborted
>>
>> Thank you for any help you can provide.
>
> Did your disks have partitions previously? That output looks a lot like
> the protective MBR partition table for a disk with GPT partitions.
>
> Could that still existing in sector 0 be confusing mdadm?
>
> I have never personally done any md raid without partitions. To me they
> just make more sense.
>
> One way to test could be to save a copy of sector 0, then overwrite sector
> 0 with zeros and then run mdadm --examine again to see if that makes a
> difference. You can always put back the saved copy of sector 0 that way.
>
> My understanding is that the default is to put the raid superblock at
> offset 4k, so it would not overwrite an existing MBR partition table.
> If it also happens due to rounding that the end of the disk isn't
> overwritten (or even just because that part of the filesystem wasn't
> written to yet), then the backup GPT from before would still be intact,
> and could perhaps cause even more confussion later if gdisk or similar
> is pointed at the disk. Really want to be sure there is no trace left
> of the partition table before using it raw for md raid.
>
> Any chance the system saved an mdadm.conf file of your setup?
>
On Thu, Mar 23, 2017 at 07:09:41PM +0100, [email protected] wrote:
> Thank you very much or your reply.
>
> I naively thought that starting without partitions would be the best
> starting point, given 3 of the disks had been in a RAID5 array
> previously (possibly with partitions, not sure), but that looks like
> a bad choice, based on some other things I've googled. Lesson learned.
>
> I have an mdadm.conf file, but it could be a remnant of my previous array.
> I've already edited it trying to get things to work, so I'm
> not sure if it was updated when I created the new array or not.
>
> I see various people online have had success in my situation using
> madadm --create /dev/md0 --assume-clean --verbose --level=10 \
> --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf
>
> Some people used --assume-clean, and some didn't. Given my array wasn't done
> with its resync, maybe I should leave that out.
>
> If that would work, I guess then I need to get the data off the array,
> delete it, and recreate it with disk partitions, or risk this happening
> again at the next reboot, for whatever reason.
>
> Anyone think it's a bad idea to try mdadm --create at this point?
>
> Sorry, I'm not sure how to write 0's to sector 0...
Well the other possibility is that by having a MBR saying this has GPT and
probably having mdadm corrupt the GPT by writing to 4k in, but nothing
overwriting the backup GPT at the end of the device, something may have
restored the GPT to the original location, even if it was an empty GPT,
in which case your superblock is likely overwritten.
Does 'gdisk -l /dev/sdc' claim that it is a valid GPT table or corrupt?
So if that is what happened (not sure what would have done it, although
it seems badly defined who is responsible for fixing the GPT if it is
detected that the backup is valid but the primary is corrupt), then you
will want to make sure the MBR and GPT tables are removed, before you
do anything else. Likely the data is still there, since I think the
first data block is offset a bit into the device, and the GPT is only
the first 32KB or so of the device. If that is the case, doing the
create would probably work. Of course the safest thing to do would be
to clone your disks before trying to reassemble them (which means you
need another four 4TB drives, but if the data is important, that's
probably worth it).
Of course you would also have to be sure the mdadm command you use to
create it again is exactly the same as before, and that the device names
are exactly the same as before.
--
Len Sorensen
Apologies, I should have started this on linux-raid...
stephen@fred> sudo gdisk -l /dev/sdc
GPT fdisk (gdisk) version 1.0.1
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdc: 7814037168 sectors, 3.6 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)
Number Start (sector) End (sector) Size Code Name
stephen@fred>
The other disks are all similar.
How do I remove the MBR/GPT tables? So you would do that, and then the
mdadm --create?
Thank you.
On 3/23/2017 20:27, Lennart Sorensen wrote:
> On Thu, Mar 23, 2017 at 07:09:41PM +0100, [email protected] wrote:
>> Thank you very much or your reply.
>>
>> I naively thought that starting without partitions would be the best
>> starting point, given 3 of the disks had been in a RAID5 array
>> previously (possibly with partitions, not sure), but that looks like
>> a bad choice, based on some other things I've googled. Lesson learned.
>>
>> I have an mdadm.conf file, but it could be a remnant of my previous array.
>> I've already edited it trying to get things to work, so I'm
>> not sure if it was updated when I created the new array or not.
>>
>> I see various people online have had success in my situation using
>> madadm --create /dev/md0 --assume-clean --verbose --level=10 \
>> --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf
>>
>> Some people used --assume-clean, and some didn't. Given my array wasn't done
>> with its resync, maybe I should leave that out.
>>
>> If that would work, I guess then I need to get the data off the array,
>> delete it, and recreate it with disk partitions, or risk this happening
>> again at the next reboot, for whatever reason.
>>
>> Anyone think it's a bad idea to try mdadm --create at this point?
>>
>> Sorry, I'm not sure how to write 0's to sector 0...
>
> Well the other possibility is that by having a MBR saying this has GPT and
> probably having mdadm corrupt the GPT by writing to 4k in, but nothing
> overwriting the backup GPT at the end of the device, something may have
> restored the GPT to the original location, even if it was an empty GPT,
> in which case your superblock is likely overwritten.
>
> Does 'gdisk -l /dev/sdc' claim that it is a valid GPT table or corrupt?
>
> So if that is what happened (not sure what would have done it, although
> it seems badly defined who is responsible for fixing the GPT if it is
> detected that the backup is valid but the primary is corrupt), then you
> will want to make sure the MBR and GPT tables are removed, before you
> do anything else. Likely the data is still there, since I think the
> first data block is offset a bit into the device, and the GPT is only
> the first 32KB or so of the device. If that is the case, doing the
> create would probably work. Of course the safest thing to do would be
> to clone your disks before trying to reassemble them (which means you
> need another four 4TB drives, but if the data is important, that's
> probably worth it).
>
> Of course you would also have to be sure the mdadm command you use to
> create it again is exactly the same as before, and that the device names
> are exactly the same as before.
>
On Thu, Mar 23, 2017 at 08:38:08PM +0100, Stephen Mueller wrote:
> Apologies, I should have started this on linux-raid...
>
>
> stephen@fred> sudo gdisk -l /dev/sdc
> GPT fdisk (gdisk) version 1.0.1
>
> Partition table scan:
> MBR: protective
> BSD: not present
> APM: not present
> GPT: present
>
> Found valid GPT with protective MBR; using GPT.
> Disk /dev/sdc: 7814037168 sectors, 3.6 TiB
> Logical sector size: 512 bytes
> Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB
> Partition table holds up to 128 entries
> First usable sector is 34, last usable sector is 7814037134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 7814037101 sectors (3.6 TiB)
>
> Number Start (sector) End (sector) Size Code Name
> stephen@fred>
>
> The other disks are all similar.
>
> How do I remove the MBR/GPT tables? So you would do that, and then the
> mdadm --create?
Well at least that seems to confirm that something restored the GPT from
the second copy at the end of the disk, wiping out the md superblock
at 4k.
If you run gdisk /dev/sdc, it has a 'zap' option using the z key to
delete all traces of GPT. That ought to do what you want.
--
Len Sorensen
OK, I used gdisk to remove the GPT and MBR from each disk.
mdadm --assemble still doesn't work... says it can't find the
superblock. The mdadm --examine commands also say that no
superblock is detected.
I guess I'll go ahead with --create...
On 3/23/2017 20:59, Lennart Sorensen wrote:
> On Thu, Mar 23, 2017 at 08:38:08PM +0100, Stephen Mueller wrote:
>> Apologies, I should have started this on linux-raid...
>>
>>
>> stephen@fred> sudo gdisk -l /dev/sdc
>> GPT fdisk (gdisk) version 1.0.1
>>
>> Partition table scan:
>> MBR: protective
>> BSD: not present
>> APM: not present
>> GPT: present
>>
>> Found valid GPT with protective MBR; using GPT.
>> Disk /dev/sdc: 7814037168 sectors, 3.6 TiB
>> Logical sector size: 512 bytes
>> Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB
>> Partition table holds up to 128 entries
>> First usable sector is 34, last usable sector is 7814037134
>> Partitions will be aligned on 2048-sector boundaries
>> Total free space is 7814037101 sectors (3.6 TiB)
>>
>> Number Start (sector) End (sector) Size Code Name
>> stephen@fred>
>>
>> The other disks are all similar.
>>
>> How do I remove the MBR/GPT tables? So you would do that, and then the
>> mdadm --create?
>
> Well at least that seems to confirm that something restored the GPT from
> the second copy at the end of the disk, wiping out the md superblock
> at 4k.
>
> If you run gdisk /dev/sdc, it has a 'zap' option using the z key to
> delete all traces of GPT. That ought to do what you want.
>
On Thu, Mar 23, 2017 at 09:09:39PM +0100, Stephen Mueller wrote:
> OK, I used gdisk to remove the GPT and MBR from each disk.
> mdadm --assemble still doesn't work... says it can't find the
> superblock. The mdadm --examine commands also say that no
> superblock is detected.
Yes the GPT overwrite the superblock. The GPT uses the first 17KB of
the disk, and the superblock was at 4KB from the start.
> I guess I'll go ahead with --create...
At least given the data should start at 1MB from the start, it should
not have been overwritten.
I do always like disk images before I try anything like that. I am never
quite sure what create will do although it will likely do the right
thing at least if you get the order right.
--
Len Sorensen
Looks like it worked! Thanks!
I used:
sudo mdadm --create /dev/md0 --assume-clean --verbose --level-10
--raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf
And I got my instructions for creating the array here, and they
also don't use partitions...
https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04
On 3/23/2017 21:09, Stephen Mueller wrote:
> OK, I used gdisk to remove the GPT and MBR from each disk.
> mdadm --assemble still doesn't work... says it can't find the
> superblock. The mdadm --examine commands also say that no
> superblock is detected.
>
> I guess I'll go ahead with --create...
>
>
> On 3/23/2017 20:59, Lennart Sorensen wrote:
>> On Thu, Mar 23, 2017 at 08:38:08PM +0100, Stephen Mueller wrote:
>>> Apologies, I should have started this on linux-raid...
>>>
>>>
>>> stephen@fred> sudo gdisk -l /dev/sdc
>>> GPT fdisk (gdisk) version 1.0.1
>>>
>>> Partition table scan:
>>> MBR: protective
>>> BSD: not present
>>> APM: not present
>>> GPT: present
>>>
>>> Found valid GPT with protective MBR; using GPT.
>>> Disk /dev/sdc: 7814037168 sectors, 3.6 TiB
>>> Logical sector size: 512 bytes
>>> Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB
>>> Partition table holds up to 128 entries
>>> First usable sector is 34, last usable sector is 7814037134
>>> Partitions will be aligned on 2048-sector boundaries
>>> Total free space is 7814037101 sectors (3.6 TiB)
>>>
>>> Number Start (sector) End (sector) Size Code Name
>>> stephen@fred>
>>>
>>> The other disks are all similar.
>>>
>>> How do I remove the MBR/GPT tables? So you would do that, and then the
>>> mdadm --create?
>>
>> Well at least that seems to confirm that something restored the GPT from
>> the second copy at the end of the disk, wiping out the md superblock
>> at 4k.
>>
>> If you run gdisk /dev/sdc, it has a 'zap' option using the z key to
>> delete all traces of GPT. That ought to do what you want.
>>
On Thu, Mar 23, 2017 at 10:02:23PM +0100, Stephen Mueller wrote:
> Looks like it worked! Thanks!
Well at least you could backup the data, just in case.
> I used:
>
> sudo mdadm --create /dev/md0 --assume-clean --verbose --level-10
> --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf
I wonder about the assume-clean, but I would think any data you wrote
will have been duplicated at the time you wrote it, and only unused
space might not have been synced yet.
Maybe running a forced resync would be worthwhile.
> And I got my instructions for creating the array here, and they
> also don't use partitions...
>
> https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04
Well it works, as long as you don't leave anything around to confuse it,
in this case a GPT partition table that somehow came back to bite you.
I wonder if mdadm could perhaps warn about the existing partition table
when being asked to create a new device.
Does it survive reboots now?
--
Len Sorensen
First I'm getting the data off the RAID... then I'm going to delete the
whole thing again... create a new RAID using partitions... follow every
step carefully... then once the new RAID array is there, I'll throw a
bit of data on it, and then reboot and see if it's still there...
if so, I'll repopulate all the data. Thanks.
On 3/23/2017 22:09, Lennart Sorensen wrote:
> On Thu, Mar 23, 2017 at 10:02:23PM +0100, Stephen Mueller wrote:
>> Looks like it worked! Thanks!
>
> Well at least you could backup the data, just in case.
>
>> I used:
>>
>> sudo mdadm --create /dev/md0 --assume-clean --verbose --level-10
>> --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf
>
> I wonder about the assume-clean, but I would think any data you wrote
> will have been duplicated at the time you wrote it, and only unused
> space might not have been synced yet.
>
> Maybe running a forced resync would be worthwhile.
>
>> And I got my instructions for creating the array here, and they
>> also don't use partitions...
>>
>> https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04
>
> Well it works, as long as you don't leave anything around to confuse it,
> in this case a GPT partition table that somehow came back to bite you.
>
> I wonder if mdadm could perhaps warn about the existing partition table
> when being asked to create a new device.
>
> Does it survive reboots now?
>