2016-12-13 15:39:09

by Heinz Nimmervoll

Subject: superblock completely overwritten

I still got no answer for my problem thats why I try it here... hopefully you could help me out.
 
System:

Embedded board with Atmel SAM9x25
Debian Wheezy Kernel 3.11.6
32GB Samsung SDHC card with ext4 root- partition (journal activated)
 

After system running two weeks or so superblock from rootfs (ext4) at block 0 got overwritten with "trash data".
This is happening with like 20% of the embedded devices.
 
hex comparision between faulty and good superblock starting at byte 1024:
 
Before (good):
 
00000000  00 ee 02 00 00 b8 0b 00  00 96 00 00 ab a9 05 00  |................|
00000010  c3 0c 02 00 00 00 00 00  02 00 00 00 02 00 00 00  |................|
00000020  00 80 00 00 00 80 00 00  40 1f 00 00 9e 68 46 58  |[email protected]|
00000030  9e 68 46 58 2e 00 64 00  53 ef 01 00 01 00 00 00  |.hFX..d.S.......|
00000040  0f 35 ff 55 00 00 00 00  00 00 00 00 01 00 00 00  |.5.U............|
00000050  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 00 00  |............<...|
00000060  46 02 00 00 7b 00 00 00  3b 9e 9c 82 4d 90 4d ed  |F...{...;...M.M.|
00000070  84 41 3a 05 5c 37 fb a8  72 6f 6f 74 66 73 00 00  |.A:.\7..rootfs..|
00000080  00 00 00 00 00 00 00 00  2f 6d 65 64 69 61 2f 76  |......../media/v|
00000090  69 72 74 75 61 6c 73 69  67 69 2f 72 6f 6f 74 66  |irtualsigi/rootf|
000000a0  73 00 73 00 00 00 00 00  00 00 00 00 00 00 00 00  |s.s.............|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 bb 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  08 00 00 00 00 00 00 00  00 00 00 00 41 41 7f 29  |............AA.)|
000000f0  b2 43 48 94 ab 07 f8 fa  38 1c 2e 33 01 01 00 00  |.CH.....8..3....|
00000100  0c 00 00 00 00 00 00 00  b8 40 6a 52 0a f3 01 00  |.........@jR....|
00000110  04 00 00 00 00 00 00 00  00 00 00 00 00 40 00 00  |.............@..|
00000120  00 00 05 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 04  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 1c 00 1c 00  |................|
00000160  02 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 04 00 00 00  cd 8a 94 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 
After (corrupted):
 
00000000  00 00 00 00 a4 81 00 00  dd 00 00 00 24 8e 5d 54  |............$.]T|
00000010  7e 8e 5d 54 18 a6 9f 41  00 00 00 00 00 00 01 00  |~.]T...A........|
00000020  08 00 00 00 00 00 08 00  01 00 00 00 0a f3 01 00  |................|
00000030  04 00 00 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000040  54 b2 01 00 00 00 00 00  00 00 00 00 00 00 00 00  |T...............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  86 2a 63 08 00 00 00 00  |.........*c.....|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 1c 00 00 00  60 c9 e6 31 00 00 00 00  |........`..1....|
00000090  00 00 00 00 7c 8e 5d 54  9c b0 2a ec 00 00 00 00  |....|.]T..*.....|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100  00 00 00 00 a4 81 00 00  0f 00 00 00 67 4d 6b 52  |............gMkR|
00000110  67 4d 6b 52 63 87 84 4f  00 00 00 00 00 00 01 00  |gMkRc..O........|
00000120  08 00 00 00 00 00 08 00  01 00 00 00 0a f3 01 00  |................|
00000130  04 00 00 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000140  74 07 03 00 00 00 00 00  00 00 00 00 00 00 00 00  |t...............|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  19 1e 79 cd 00 00 00 00  |..........y.....|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 1c 00 00 00  e4 14 dc e7 00 00 00 00  |................|
00000190  98 6a 83 72 67 4d 6b 52  98 6a 83 72 00 00 00 00  |.j.rgMkR.j.r....|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  00 00 00 00 a4 81 00 00  88 0c 00 00 44 8c 5d 54  |............D.]T|
00000210  4d 8c 5d 54 9c 8e 52 54  00 00 00 00 00 00 01 00  |M.]T..RT........|
00000220  08 00 00 00 00 00 08 00  01 00 00 00 0a f3 01 00  |................|
00000230  04 00 00 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000240  7b e0 01 00 00 00 00 00  00 00 00 00 00 00 00 00  |{...............|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  4e 09 63 08 00 00 00 00  |........N.c.....|
00000270  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 1c 00 00 00  f8 b4 6f 14 00 00 00 00  |..........o.....|
00000290  00 00 00 00 48 8c 5d 54  28 cc 55 82 00 00 00 00  |....H.]T(.U.....|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000300  00 00 00 00 a4 81 00 00  94 05 00 00 68 4d 6b 52  |............hMkR|
00000310  69 4d 6b 52 54 0f df 4f  00 00 00 00 00 00 01 00  |iMkRT..O........|
00000320  08 00 00 00 00 00 08 00  01 00 00 00 0a f3 01 00  |................|
00000330  04 00 00 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000340  8c 07 03 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000350  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000360  00 00 00 00 00 00 00 00  27 1e 79 cd 00 00 00 00  |........'.y.....|
00000370  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000380  00 00 00 00 1c 00 00 00  b0 07 a2 3a 00 00 00 00  |...........:....|
00000390  04 a9 de 29 68 4d 6b 52  04 a9 de 29 00 00 00 00  |...)hMkR...)....|
000003a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

- How is it possible, that even the magic number (and everything else) got overwritten?
- Why could it ever be overwritten?
 
Thank you so much!

2016-12-13 16:16:22

by Eric Sandeen

Subject: Re: superblock completely overwritten

On 12/13/16 9:39 AM, Heinz Nimmervoll wrote:
> I still got no answer for my problem thats why I try it here... hopefully you could help me out.
>
> System:
>
> Embedded board with Atmel SAM9x25
> Debian Wheezy Kernel 3.11.6
> 32GB Samsung SDHC card with ext4 root- partition (journal activated)
>
>
> After system running two weeks or so superblock from rootfs (ext4) at block 0 got overwritten with "trash data".
> This is happening with like 20% of the embedded devices.
>
> hex comparision between faulty and good superblock starting at byte 1024:
>
> Before (good):
>
> 00000000 00 ee 02 00 00 b8 0b 00 00 96 00 00 ab a9 05 00 |................|
> 00000010 c3 0c 02 00 00 00 00 00 02 00 00 00 02 00 00 00 |................|
> 00000020 00 80 00 00 00 80 00 00 40 1f 00 00 9e 68 46 58 |[email protected]|
> 00000030 9e 68 46 58 2e 00 64 00 53 ef 01 00 01 00 00 00 |.hFX..d.S.......|

<snip>

>
> After (corrupted):
>
> 00000000 00 00 00 00 a4 81 00 00 dd 00 00 00 24 8e 5d 54 |............$.]T|
> 00000010 7e 8e 5d 54 18 a6 9f 41 00 00 00 00 00 00 01 00 |~.]T...A........|
> 00000020 08 00 00 00 00 00 08 00 01 00 00 00 0a f3 01 00 |................|
> 00000030 04 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|


<snip>

> - How is it possible, that even the magic number (and everything else) got overwritten?
> - Why could it ever be overwritten?

I don't think anyone here can tell you what happened, it is almost certainly not
an ext4 bug. Could be a driver bug, or an admin running a stray "dd" command,
or some other utility gone astray, or ... anything, really.

As a very long shot, what does "blkid" or "file -s" tell you about the block device
after it's been overwritten? Perhaps it will recognize a signature.

Otherwise, you could do something like a modified kernel to trap any IO to block
zero on the device and issue a printk about the process which is doing it, filtering
out any expected ext4 accesses.

-Eric

> Thank you so much!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2016-12-13 18:13:55

by Darrick J. Wong

Subject: Re: superblock completely overwritten

On Tue, Dec 13, 2016 at 10:15:25AM -0600, Eric Sandeen wrote:
> On 12/13/16 9:39 AM, Heinz Nimmervoll wrote:
> > I still got no answer for my problem thats why I try it here... hopefully you could help me out.
> >
> > System:
> >
> > Embedded board with Atmel SAM9x25
> > Debian Wheezy Kernel 3.11.6
> > 32GB Samsung SDHC card with ext4 root- partition (journal activated)
> >
> >
> > After system running two weeks or so superblock from rootfs (ext4) at block 0 got overwritten with "trash data".
> > This is happening with like 20% of the embedded devices.
> >
> > hex comparision between faulty and good superblock starting at byte 1024:
> >
> > Before (good):
> >
> > 00000000 00 ee 02 00 00 b8 0b 00 00 96 00 00 ab a9 05 00 |................|
> > 00000010 c3 0c 02 00 00 00 00 00 02 00 00 00 02 00 00 00 |................|
> > 00000020 00 80 00 00 00 80 00 00 40 1f 00 00 9e 68 46 58 |[email protected]|
> > 00000030 9e 68 46 58 2e 00 64 00 53 ef 01 00 01 00 00 00 |.hFX..d.S.......|
>
> <snip>
>
> >
> > After (corrupted):
> >
> > 00000000 00 00 00 00 a4 81 00 00 dd 00 00 00 24 8e 5d 54 |............$.]T|
> > 00000010 7e 8e 5d 54 18 a6 9f 41 00 00 00 00 00 00 01 00 |~.]T...A........|
> > 00000020 08 00 00 00 00 00 08 00 01 00 00 00 0a f3 01 00 |................|
> > 00000030 04 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|

81a4? f30a? That looks like an inode block (i_mode 0x81a4) and an
extent tree root (eh_magic 0xf30a), shifted up by 4 bytes before being
written onto sector 0.

> <snip>
>
> > - How is it possible, that even the magic number (and everything else) got overwritten?
> > - Why could it ever be overwritten?
>
> I don't think anyone here can tell you what happened, it is almost certainly not
> an ext4 bug. Could be a driver bug, or an admin running a stray "dd" command,
> or some other utility gone astray, or ... anything, really.

Bad/malicious SDHC card would be my guess? (Are you doing power fail
testing?)

--D

> As a very long shot, what does "blkid" or "file -s" tell you about the block device
> after it's been overwritten? Perhaps it will recognize a signature.
>
> Otherwise, you could do something like a modified kernel to trap any IO to block
> zero on the device and issue a printk about the process which is doing it, filtering
> out any expected ext4 accesses.
>
> -Eric
>
> > Thank you so much!
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2016-12-14 09:41:31

by Heinz Nimmervoll

Subject: Aw: Re: superblock completely overwritten

> > I still got no answer for my problem thats why I try it here... hopefully you could help me out.
> >
> > System:
> >
> > Embedded board with Atmel SAM9x25
> > Debian Wheezy Kernel 3.11.6
> > 32GB Samsung SDHC card with ext4 root- partition (journal activated)
> >
> >
> > After system running two weeks or so superblock from rootfs (ext4) at block 0 got overwritten with "trash data".
> > This is happening with like 20% of the embedded devices.
> >
> > hex comparision between faulty and good superblock starting at byte 1024:
> >
> > Before (good):
> >
> > 00000000 00 ee 02 00 00 b8 0b 00 00 96 00 00 ab a9 05 00 |................|
> > 00000010 c3 0c 02 00 00 00 00 00 02 00 00 00 02 00 00 00 |................|
> > 00000020 00 80 00 00 00 80 00 00 40 1f 00 00 9e 68 46 58 |[email protected]|
> > 00000030 9e 68 46 58 2e 00 64 00 53 ef 01 00 01 00 00 00 |.hFX..d.S.......|
>
> <snip>
>
> >
> > After (corrupted):
> >
> > 00000000 00 00 00 00 a4 81 00 00 dd 00 00 00 24 8e 5d 54 |............$.]T|
> > 00000010 7e 8e 5d 54 18 a6 9f 41 00 00 00 00 00 00 01 00 |~.]T...A........|
> > 00000020 08 00 00 00 00 00 08 00 01 00 00 00 0a f3 01 00 |................|
> > 00000030 04 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|

> 81a4? f30a? That looks like an inode block (i_mode 0x81a4) and an
> extent tree root (eh_magic 0xf30a), shifted up by 4 bytes before being
> written onto sector 0.

Thx for your replys! Wow, thats interesting. Here is another hex dump from another corrupted card (as mentioned it is a reproducible problem):

00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000040 00 00 00 00 00 00 00 00 a4 81 00 00 85 06 00 00 |................|
00000050 25 e9 0b 56 f4 42 6a 52 59 7d 15 51 00 00 00 00 |%..V.BjRY}.Q....|
00000060 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000070 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000080 01 00 00 00 7f 82 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 9a df 58 e9 |..............X.|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000000d0 00 00 00 00 5c 57 7a 77 f4 42 6a 52 b0 29 af 96 |....\Wzw.BjR.)..|
000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000140 00 00 00 00 00 00 00 00 80 81 00 00 9c 02 00 00 |................|
00000150 74 db 84 57 f4 42 6a 52 d4 30 1a 52 00 00 00 00 |t..W.BjR.0.R....|
00000160 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000170 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000180 01 00 00 00 80 82 00 00 00 00 00 00 00 00 00 00 |................|
00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001a0 00 00 00 00 00 00 00 00 00 00 00 00 9b df 58 e9 |..............X.|
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000001d0 00 00 00 00 78 93 71 17 f4 42 6a 52 b0 29 af 96 |....x.q..BjR.)..|
000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000240 00 00 00 00 00 00 00 00 a4 81 00 00 bc 00 00 00 |................|
00000250 22 4c 41 53 f4 42 6a 52 d4 30 1a 52 00 00 00 00 |"LAS.BjR.0.R....|
00000260 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000270 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000280 01 00 00 00 81 82 00 00 00 00 00 00 00 00 00 00 |................|
00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000002a0 00 00 00 00 00 00 00 00 00 00 00 00 9c df 58 e9 |..............X.|
000002b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000002c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000002d0 00 00 00 00 bc 8b 80 a8 f4 42 6a 52 b0 29 af 96 |.........BjR.)..|
000002e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000340 00 00 00 00 00 00 00 00 a4 81 00 00 dc 13 02 00 |................|
00000350 7e dd 84 57 f4 42 6a 52 59 7d 15 51 00 00 00 00 |~..W.BjRY}.Q....|
00000360 00 00 01 00 10 01 00 00 00 00 08 00 01 00 00 00 |................|
00000370 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000380 22 00 00 00 82 82 00 00 00 00 00 00 00 00 00 00 |"...............|
00000390 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000003a0 00 00 00 00 00 00 00 00 00 00 00 00 9d df 58 e9 |..............X.|
000003b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000003c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000003d0 00 00 00 00 d8 ed 4c 81 f4 42 6a 52 b0 29 af 96 |......L..BjR.)..|
000003e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000440 00 00 00 00 00 00 00 00 80 81 00 00 e3 00 00 00 |................|
00000450 74 db 84 57 f4 42 6a 52 d4 30 1a 52 00 00 00 00 |t..W.BjR.0.R....|
00000460 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000470 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000480 01 00 00 00 a4 82 00 00 00 00 00 00 00 00 00 00 |................|
00000490 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000004a0 00 00 00 00 00 00 00 00 00 00 00 00 9e df 58 e9 |..............X.|
000004b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000004c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000004d0 00 00 00 00 78 3f 2b 1b f4 42 6a 52 b0 29 af 96 |....x?+..BjR.)..|
000004e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000540 00 00 00 00 00 00 00 00 a4 81 00 00 de 1b 00 00 |................|
00000550 93 c5 fe 54 f4 42 6a 52 5f 31 1a 52 00 00 00 00 |...T.BjR_1.R....|
00000560 00 00 01 00 10 00 00 00 00 00 08 00 01 00 00 00 |................|
00000570 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000580 02 00 00 00 a6 82 00 00 00 00 00 00 00 00 00 00 |................|
00000590 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000005a0 00 00 00 00 00 00 00 00 00 00 00 00 9f df 58 e9 |..............X.|
000005b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000005c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000005d0 00 00 00 00 34 b9 b4 05 f4 42 6a 52 b0 29 af 96 |....4....BjR.)..|
000005e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000640 00 00 00 00 00 00 00 00 ed 41 00 00 00 10 00 00 |.........A......|
00000650 1d d0 bb 57 32 42 ad 53 32 42 ad 53 00 00 00 00 |...W2B.S2B.S....|
00000660 00 00 02 00 08 00 00 00 00 00 08 00 16 00 00 00 |................|
00000670 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000680 01 00 00 00 91 20 00 00 00 00 00 00 00 00 00 00 |..... ..........|
00000690 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000006a0 00 00 00 00 00 00 00 00 00 00 00 00 a0 df 58 e9 |..............X.|
000006b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000006c0 00 00 00 00 00 00 00 00 1c 00 00 00 10 ec 28 b8 |..............(.|
000006d0 10 ec 28 b8 e0 2b ac 8b f4 42 6a 52 b0 29 af 96 |..(..+...BjR.)..|
000006e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000740 00 00 00 00 00 00 00 00 ed 81 00 00 00 01 00 00 |................|
00000750 f3 d0 bb 57 f4 42 6a 52 59 b3 46 51 00 00 00 00 |...W.BjRY.FQ....|
00000760 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000770 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000780 01 00 00 00 a5 82 00 00 00 00 00 00 00 00 00 00 |................|
00000790 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000007a0 00 00 00 00 00 00 00 00 00 00 00 00 a1 df 58 e9 |..............X.|
000007b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000007c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000007d0 00 00 00 00 68 68 e2 0c f4 42 6a 52 b0 29 af 96 |....hh...BjR.)..|
000007e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000840 00 00 00 00 00 00 00 00 ed 81 00 00 79 02 00 00 |............y...|
00000850 1d d0 bb 57 f4 42 6a 52 53 21 35 51 00 00 00 00 |...W.BjRS!5Q....|
00000860 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000870 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000880 01 00 00 00 a8 82 00 00 00 00 00 00 00 00 00 00 |................|
00000890 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000008a0 00 00 00 00 00 00 00 00 00 00 00 00 a2 df 58 e9 |..............X.|
000008b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000008c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000008d0 00 00 00 00 e4 ad 42 91 f4 42 6a 52 b0 29 af 96 |......B..BjR.)..|
000008e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000940 00 00 00 00 00 00 00 00 ed 81 00 00 59 00 00 00 |............Y...|
00000950 19 2e b9 57 f4 42 6a 52 9d 10 b5 4f 00 00 00 00 |...W.BjR...O....|
00000960 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000970 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000980 01 00 00 00 a9 82 00 00 00 00 00 00 00 00 00 00 |................|
00000990 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000009a0 00 00 00 00 00 00 00 00 00 00 00 00 a3 df 58 e9 |..............X.|
000009b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000009c0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
000009d0 00 00 00 00 68 63 ea 99 f4 42 6a 52 b0 29 af 96 |....hc...BjR.)..|
000009e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000a40 00 00 00 00 00 00 00 00 ed 81 00 00 63 01 00 00 |............c...|
00000a50 f3 d0 bb 57 f4 42 6a 52 bb df d5 4f 00 00 00 00 |...W.BjR...O....|
00000a60 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000a70 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000a80 01 00 00 00 aa 82 00 00 00 00 00 00 00 00 00 00 |................|
00000a90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000aa0 00 00 00 00 00 00 00 00 00 00 00 00 a4 df 58 e9 |..............X.|
00000ab0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ac0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
00000ad0 00 00 00 00 68 bc 28 09 f4 42 6a 52 b0 29 af 96 |....h.(..BjR.)..|
00000ae0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000b40 00 00 00 00 00 00 00 00 ed 81 00 00 89 3a 00 00 |.............:..|
00000b50 1d d0 bb 57 f4 42 6a 52 77 15 b0 51 00 00 00 00 |...W.BjRw..Q....|
00000b60 00 00 01 00 20 00 00 00 00 00 08 00 01 00 00 00 |.... ...........|
00000b70 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000b80 04 00 00 00 ac 82 00 00 00 00 00 00 00 00 00 00 |................|
00000b90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ba0 00 00 00 00 00 00 00 00 00 00 00 00 a5 df 58 e9 |..............X.|
00000bb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000bc0 00 00 00 00 00 00 00 00 1c 00 00 00 b0 29 af 96 |.............)..|
00000bd0 00 00 00 00 ec b1 6f 9c f4 42 6a 52 b0 29 af 96 |......o..BjR.)..|
00000be0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000c40 00 00 00 00 00 00 00 00 a4 81 00 00 66 00 00 00 |............f...|
00000c50 f8 87 98 45 f4 42 6a 52 3e 6f f3 4f 00 00 00 00 |...E.BjR>o.O....|
00000c60 00 00 01 00 08 00 00 00 00 00 08 00 01 00 00 00 |................|
00000c70 0a f3 01 00 04 00 00 00 00 00 00 00 00 00 00 00 |................|
00000c80 01 00 00 00 ab 82 00 00 00 00 00 00 00 00 00 00 |................|
00000c90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ca0 00 00 00 00 00 00 00 00 00 00 00 00 a6 df 58 e9 |..............X.|
00000cb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000cc0 00 00 00 00 00 00 00 00 1c 00 00 00 ac 34 ec 96 |.............4..|
00000cd0 00 00 00 00 a8 cb 1c ad f4 42 6a 52 ac 34 ec 96 |.........BjR.4..|
00000ce0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000d40 00 00 00 00 00 00 00 00 ff a1 00 00 19 00 00 00 |................|
00000d50 73 75 16 56 f7 42 6a 52 70 31 1a 52 00 00 00 00 |su.V.BjRp1.R....|
00000d60 00 00 01 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|
00000d70 2e 2e 2f 69 6e 69 74 2e 64 2f 62 6f 6f 74 5f 73 |../init.d/boot_s|
00000d80 63 72 69 70 74 73 2e 73 68 00 00 00 00 00 00 00 |cripts.sh.......|
00000d90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000da0 00 00 00 00 00 00 00 00 00 00 00 00 c8 32 59 e9 |.............2Y.|
00000db0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000dc0 00 00 00 00 00 00 00 00 1c 00 00 00 30 48 96 d8 |............0H..|
00000dd0 00 00 00 00 68 a6 5a 83 f7 42 6a 52 30 48 96 d8 |....h.Z..BjR0H..|
00000de0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000e40 00 00 00 00 00 00 00 00 ff a1 00 00 0e 00 00 00 |................|
00000e50 73 75 16 56 cb a3 0a 56 cb a3 0a 56 00 00 00 00 |su.V...V...V....|
00000e60 00 00 01 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|
00000e70 2e 2e 2f 69 6e 69 74 2e 64 2f 63 72 6f 6e 00 00 |../init.d/cron..|
00000e80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000ea0 00 00 00 00 00 00 00 00 00 00 00 00 31 92 30 60 |............1.0`|
00000eb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000ec0 00 00 00 00 00 00 00 00 1c 00 00 00 e0 32 a8 1b |.............2..|
00000ed0 e0 32 a8 1b 78 ae b4 99 cb a3 0a 56 e0 32 a8 1b |.2..x......V.2..|
00000ee0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000f40 00 00 00 00 00 00 00 00 ff a1 00 00 0e 00 00 00 |................|
00000f50 73 75 16 56 cb a3 0a 56 cb a3 0a 56 00 00 00 00 |su.V...V...V....|
00000f60 00 00 01 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|
00000f70 2e 2e 2f 69 6e 69 74 2e 64 2f 63 72 6f 6e 00 00 |../init.d/cron..|
00000f80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000fa0 00 00 00 00 00 00 00 00 00 00 00 00 32 92 30 60 |............2.0`|
00000fb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000fc0 00 00 00 00 00 00 00 00 1c 00 00 00 e4 08 85 1d |................|
00000fd0 e4 08 85 1d 8c ba 3b bb cb a3 0a 56 e4 08 85 1d |......;....V....|
00000fe0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000

> <snip>
>
> > - How is it possible, that even the magic number (and everything else) got overwritten?
> > - Why could it ever be overwritten?
>
> I don't think anyone here can tell you what happened, it is almost certainly not
> an ext4 bug. Could be a driver bug, or an admin running a stray "dd" command,
> or some other utility gone astray, or ... anything, really.

I can rule out that it was the dd command. Eric, do you think it could also be a problem about the SD-card and wear levelling (just a guess)?


> Bad/malicious SDHC card would be my guess? (Are you doing power fail testing?)

We did have the problem only with the same type of SD-cards, which on the other hand we used most frequently also.
Don´t know what you mean by power fail testing: We tested power fail with our devices by frequently switching off devices (without any impact).
What we observed that the devices were dying "slowely", so it took some time till they werent accesible anymore.

Thx again!

2016-12-14 16:35:14

by Eric Sandeen

Subject: Re: Aw: Re: superblock completely overwritten

On 12/14/16 3:41 AM, Heinz Nimmervoll wrote:
>>> - How is it possible, that even the magic number (and everything else) got overwritten?
>>> - Why could it ever be overwritten?

>> I don't think anyone here can tell you what happened, it is almost certainly not
>> an ext4 bug. Could be a driver bug, or an admin running a stray "dd" command,
>> or some other utility gone astray, or ... anything, really.

> I can rule out that it was the dd command. Eric, do you think it
> could also be a problem about the SD-card and wear levelling (just a
> guess)?

Ok, now that Darrick has pointed out that it's inode data, I am slightly
less sure that it is "almost certainly not an ext4 bug" :)

Still, I don't recall any other reports of this type of corruption, so
I think it's quite possible that it's a problem outside of ext4.

Depending on how hard it is to reproduce, you might consider retesting
several sd card types to see if it is hardware-dependent.

-Eric