Hi,
Say I started with 1 disk using LVM with an ext4 partition.
I then added another disk. Added it to the LVM group, expanded the
ext4 partition to then fill 2 disks.
I then added another disk. Added it to the LVM group, expanded the
ext4 partition to then fill 3 disks.
One of the disk has now failed.
Are there any tools available for ext4 that could help recover this?
Note, I am a single user, not a company, so there is no money to give
to a data recovery company, so I wish to try myself.
Is there a mount option that will mount an ext4 partition even if
there are lots of sector errors, or a missing disk. So, at least some
of the files will be recoverable.
Or a tool that will scan the disk for super blocks, then find inodes
and then find the sector location of the files, even if they don't
have filenames.
Kind Regards
James
On Jan 14, 2020, at 9:03 AM, James Courtier-Dutton <[email protected]> wrote:
>
> Hi,
>
> Say I started with 1 disk using LVM with an ext4 partition.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 2 disks.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 3 disks.
>
> One of the disk has now failed.
> Are there any tools available for ext4 that could help recover this?
> Note, I am a single user, not a company, so there is no money to give
> to a data recovery company, so I wish to try myself.
>
> Is there a mount option that will mount an ext4 partition even if
> there are lots of sector errors, or a missing disk. So, at least some
> of the files will be recoverable.
> Or a tool that will scan the disk for super blocks, then find inodes
> and then find the sector location of the files, even if they don't
> have filenames.
The first thing I would do is get new disk(s) to make a full backup of
the existing disks, so that whatever you do for recovery doesn't cause
further data loss.
There are two approaches to recovering the data:
- add in a replacement disk and re-add the missing LVs that fill in the
lost space of the failed disk, then run e2fsck to rebuild the metadata
for the missing disk. I'm not sure how easy/hard this is with LVM.
- use a block-level scanner to recover files on disk, which is useful for
e.g. .jpg images, or other files with good magic signatures and internal
data integrity checks.
You didn't mention whether it was the first, second, or third disk that
failed. If it is the third disk, that may not even need to be replaced,
since you could set the filesystem size to be the end of the second disk
and then e2fsck will repair the bad block/inode references. If it is the
first or second disk then it is more complex.
Cheers, Andreas
On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
>
> Say I started with 1 disk using LVM with an ext4 partition.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 2 disks.
> I then added another disk. Added it to the LVM group, expanded the
> ext4 partition to then fill 3 disks.
Where you using RAID 0, or some more advanced RAID level?
> One of the disk has now failed.
How has it failed? It is dead dead dead? Or are there a large number
of sector errors?
> Are there any tools available for ext4 that could help recover this?
> Note, I am a single user, not a company, so there is no money to give
> to a data recovery company, so I wish to try myself.
How valuable is your data? The first thing I would recommend, if your
data is worth it (and only you can make that decision) is to create a
new RAID set (using larger disks if that helps reduce the price) so
you can make an block-level image backup using the dd_rescue program.
If you can, then run e2fsck on the backup copy, and then see what you
can recover from that image set. This will save time (how much is
your time worth?) and perhaps increase the amount of data you can
recover (how much is your data worth?).
- Ted
On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <[email protected]> wrote:
>
> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
> >
> > Say I started with 1 disk using LVM with an ext4 partition.
> > I then added another disk. Added it to the LVM group, expanded the
> > ext4 partition to then fill 2 disks.
> > I then added another disk. Added it to the LVM group, expanded the
> > ext4 partition to then fill 3 disks.
>
> Where you using RAID 0, or some more advanced RAID level?
>
> > One of the disk has now failed.
>
> How has it failed? It is dead dead dead? Or are there a large number
> of sector errors?
>
> > Are there any tools available for ext4 that could help recover this?
> > Note, I am a single user, not a company, so there is no money to give
> > to a data recovery company, so I wish to try myself.
>
> How valuable is your data? The first thing I would recommend, if your
> data is worth it (and only you can make that decision) is to create a
> new RAID set (using larger disks if that helps reduce the price) so
> you can make an block-level image backup using the dd_rescue program.
>
> If you can, then run e2fsck on the backup copy, and then see what you
> can recover from that image set. This will save time (how much is
> your time worth?) and perhaps increase the amount of data you can
> recover (how much is your data worth?).
>
> - Ted
Hi,
Thank you all for the help.
I have made some progress.
Disk 1 is 100% OK.
Disk 2 did dd_rescue and recovered about 30% of the sectors.
Disk 3 is 100% OK.
I have made images of all of that. LVM worked out what order they went
in, and ext4 even let me mount the LVM volume.
I have managed to recover a lot of the data, so thank you for the hints.
What I would like to do now is find out which files were on Disk 2.
I have found the fiemap IOCTL that gives me the logical sectors for
the file itself.
How do I convert the value from the IOCTL into a physical sector, or
at least an offset into the LVM volume?
I can then work at that sector 0 to A is on Disk 1, A to B is on Disk
2, and B to C is on Disk 3.
For example, this is how it is done on btrfs. What is the equivalent on ext4?
$ sudo filefrag -v UEFI_Spec_2_8_final.pdf
Filesystem type is: 9123683e
File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 4537: 9798022.. 9802559: 4538: last,eof
UEFI_Spec_2_8_final.pdf: 1 extent found
$ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7
mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7
$ sudo dd if=/dev/nvme0n1p7 bs=1
skip=31475654656 count=64 2>/dev/null | hexdump -C
00000000 25 50 44 46 2d 31 2e 36 0d 25 e2 e3 cf d3 0d 0a |%PDF-1.6.%......|
00000010 33 30 35 35 31 39 20 30 20 6f 62 6a 0d 3c 3c 2f |305519 0 obj.<</|
00000020 46 69 6c 74 65 72 2f 46 6c 61 74 65 44 65 63 6f |Filter/FlateDeco|
00000030 64 65 2f 46 69 72 73 74 20 31 31 37 39 2f 4c 65 |de/First 1179/Le|
00000040
Kind Regards
James
You can use "filefrag -k" to get units of KB instead of blocks for the physical offsets of the files, and "lvdisplay -m /dev/XXX" to show the mapping of LVs to disks (not sure if you can get it to print different units, or just LE number and you gave to multiply by the LE size). Hopefully the LV was layer out linearly and is just a matter of computing "> X and < Y" to determine the affected range.
Note that it is likely that some files which were on the second disk will not show anything from filefrag since the inodes were zeroed out by dd_rescue. Some files on the second disk that _do_ show blocks that were allocated on that disk may just have zeroes for the data.
Cheers, Andreas
> On Jan 20, 2020, at 15:07, James Courtier-Dutton <[email protected]> wrote:
>
> On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <[email protected]> wrote:
>>
>>> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
>>>
>>> Say I started with 1 disk using LVM with an ext4 partition.
>>> I then added another disk. Added it to the LVM group, expanded the
>>> ext4 partition to then fill 2 disks.
>>> I then added another disk. Added it to the LVM group, expanded the
>>> ext4 partition to then fill 3 disks.
>>
>> Where you using RAID 0, or some more advanced RAID level?
>>
>>> One of the disk has now failed.
>>
>> How has it failed? It is dead dead dead? Or are there a large number
>> of sector errors?
>>
>>> Are there any tools available for ext4 that could help recover this?
>>> Note, I am a single user, not a company, so there is no money to give
>>> to a data recovery company, so I wish to try myself.
>>
>> How valuable is your data? The first thing I would recommend, if your
>> data is worth it (and only you can make that decision) is to create a
>> new RAID set (using larger disks if that helps reduce the price) so
>> you can make an block-level image backup using the dd_rescue program.
>>
>> If you can, then run e2fsck on the backup copy, and then see what you
>> can recover from that image set. This will save time (how much is
>> your time worth?) and perhaps increase the amount of data you can
>> recover (how much is your data worth?).
>>
>> - Ted
>
> Hi,
>
> Thank you all for the help.
> I have made some progress.
> Disk 1 is 100% OK.
> Disk 2 did dd_rescue and recovered about 30% of the sectors.
> Disk 3 is 100% OK.
>
> I have made images of all of that. LVM worked out what order they went
> in, and ext4 even let me mount the LVM volume.
> I have managed to recover a lot of the data, so thank you for the hints.
> What I would like to do now is find out which files were on Disk 2.
> I have found the fiemap IOCTL that gives me the logical sectors for
> the file itself.
> How do I convert the value from the IOCTL into a physical sector, or
> at least an offset into the LVM volume?
> I can then work at that sector 0 to A is on Disk 1, A to B is on Disk
> 2, and B to C is on Disk 3.
>
> For example, this is how it is done on btrfs. What is the equivalent on ext4?
>
> $ sudo filefrag -v UEFI_Spec_2_8_final.pdf
> Filesystem type is: 9123683e
> File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes)
> ext: logical_offset: physical_offset: length: expected: flags:
> 0: 0.. 4537: 9798022.. 9802559: 4538: last,eof
> UEFI_Spec_2_8_final.pdf: 1 extent found
> $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7
> mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7
> $ sudo dd if=/dev/nvme0n1p7 bs=1
> skip=31475654656 count=64 2>/dev/null | hexdump -C
> 00000000 25 50 44 46 2d 31 2e 36 0d 25 e2 e3 cf d3 0d 0a |%PDF-1.6.%......|
> 00000010 33 30 35 35 31 39 20 30 20 6f 62 6a 0d 3c 3c 2f |305519 0 obj.<</|
> 00000020 46 69 6c 74 65 72 2f 46 6c 61 74 65 44 65 63 6f |Filter/FlateDeco|
> 00000030 64 65 2f 46 69 72 73 74 20 31 31 37 39 2f 4c 65 |de/First 1179/Le|
> 00000040
>
>
> Kind Regards
>
> James
HI,
Just an update. I managed to recover all the data I needed to. Many
thanks to everyone who responded.
Ext4 is the greatest file system!!!
Kind Regards
James
On Tue, 21 Jan 2020 at 02:20, Andreas Dilger <[email protected]> wrote:
>
> You can use "filefrag -k" to get units of KB instead of blocks for the physical offsets of the files, and "lvdisplay -m /dev/XXX" to show the mapping of LVs to disks (not sure if you can get it to print different units, or just LE number and you gave to multiply by the LE size). Hopefully the LV was layer out linearly and is just a matter of computing "> X and < Y" to determine the affected range.
>
> Note that it is likely that some files which were on the second disk will not show anything from filefrag since the inodes were zeroed out by dd_rescue. Some files on the second disk that _do_ show blocks that were allocated on that disk may just have zeroes for the data.
>
> Cheers, Andreas
>
> > On Jan 20, 2020, at 15:07, James Courtier-Dutton <[email protected]> wrote:
> >
> > On Tue, 14 Jan 2020 at 21:25, Theodore Y. Ts'o <[email protected]> wrote:
> >>
> >>> On Tue, Jan 14, 2020 at 04:03:53PM +0000, James Courtier-Dutton wrote:
> >>>
> >>> Say I started with 1 disk using LVM with an ext4 partition.
> >>> I then added another disk. Added it to the LVM group, expanded the
> >>> ext4 partition to then fill 2 disks.
> >>> I then added another disk. Added it to the LVM group, expanded the
> >>> ext4 partition to then fill 3 disks.
> >>
> >> Where you using RAID 0, or some more advanced RAID level?
> >>
> >>> One of the disk has now failed.
> >>
> >> How has it failed? It is dead dead dead? Or are there a large number
> >> of sector errors?
> >>
> >>> Are there any tools available for ext4 that could help recover this?
> >>> Note, I am a single user, not a company, so there is no money to give
> >>> to a data recovery company, so I wish to try myself.
> >>
> >> How valuable is your data? The first thing I would recommend, if your
> >> data is worth it (and only you can make that decision) is to create a
> >> new RAID set (using larger disks if that helps reduce the price) so
> >> you can make an block-level image backup using the dd_rescue program.
> >>
> >> If you can, then run e2fsck on the backup copy, and then see what you
> >> can recover from that image set. This will save time (how much is
> >> your time worth?) and perhaps increase the amount of data you can
> >> recover (how much is your data worth?).
> >>
> >> - Ted
> >
> > Hi,
> >
> > Thank you all for the help.
> > I have made some progress.
> > Disk 1 is 100% OK.
> > Disk 2 did dd_rescue and recovered about 30% of the sectors.
> > Disk 3 is 100% OK.
> >
> > I have made images of all of that. LVM worked out what order they went
> > in, and ext4 even let me mount the LVM volume.
> > I have managed to recover a lot of the data, so thank you for the hints.
> > What I would like to do now is find out which files were on Disk 2.
> > I have found the fiemap IOCTL that gives me the logical sectors for
> > the file itself.
> > How do I convert the value from the IOCTL into a physical sector, or
> > at least an offset into the LVM volume?
> > I can then work at that sector 0 to A is on Disk 1, A to B is on Disk
> > 2, and B to C is on Disk 3.
> >
> > For example, this is how it is done on btrfs. What is the equivalent on ext4?
> >
> > $ sudo filefrag -v UEFI_Spec_2_8_final.pdf
> > Filesystem type is: 9123683e
> > File size of UEFI_Spec_2_8_final.pdf is 18586279 (4538 blocks of 4096 bytes)
> > ext: logical_offset: physical_offset: length: expected: flags:
> > 0: 0.. 4537: 9798022.. 9802559: 4538: last,eof
> > UEFI_Spec_2_8_final.pdf: 1 extent found
> > $ sudo btrfs-map-logical -l $[9798022*4096] /dev/nvme0n1p7
> > mirror 1 logical 40132698112 physical 31475654656 device /dev/nvme0n1p7
> > $ sudo dd if=/dev/nvme0n1p7 bs=1
> > skip=31475654656 count=64 2>/dev/null | hexdump -C
> > 00000000 25 50 44 46 2d 31 2e 36 0d 25 e2 e3 cf d3 0d 0a |%PDF-1.6.%......|
> > 00000010 33 30 35 35 31 39 20 30 20 6f 62 6a 0d 3c 3c 2f |305519 0 obj.<</|
> > 00000020 46 69 6c 74 65 72 2f 46 6c 61 74 65 44 65 63 6f |Filter/FlateDeco|
> > 00000030 64 65 2f 46 69 72 73 74 20 31 31 37 39 2f 4c 65 |de/First 1179/Le|
> > 00000040
> >
> >
> > Kind Regards
> >
> > James