From: "Heinz Nimmervoll" <bt1now@gmx.at>
Subject: Aw: Re: superblock completely overwritten
Date: Wed, 14 Dec 2016 10:41:19 +0100
Message-ID: <trinity-6030b51c-4ef2-49de-bc0d-da61db362d9c-1481708479043@3capp-gmx-bs37>
References: <trinity-fba1c40f-e300-43ee-a4cd-f9d7f48ecef2-1481643547528@3capp-gmx-bs76>
 <7a0b791c-8a9f-23e5-02a2-cd903024ef06@redhat.com>,
 <20161213181346.GA6354@birch.djwong.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8BIT
Cc: "Eric Sandeen" <sandeen@redhat.com>, linux-ext4@vger.kernel.org
To: "Darrick J. Wong" <darrick.wong@oracle.com>
Return-path: <linux-ext4-owner@vger.kernel.org>
Received: from mout.gmx.net ([212.227.17.22]:49980 "EHLO mout.gmx.net"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1755165AbcLNJlb (ORCPT <rfc822;linux-ext4@vger.kernel.org>);
        Wed, 14 Dec 2016 04:41:31 -0500
In-Reply-To: <20161213181346.GA6354@birch.djwong.org>
Sender: linux-ext4-owner@vger.kernel.org
List-ID: <linux-ext4.vger.kernel.org>

> > I still got no answer for my problem thats why I try it here... hopefully you could help me out.
> >
> > System:
> >
> > Embedded board with Atmel SAM9x25
> > Debian Wheezy Kernel 3.11.6
> > 32GB Samsung SDHC card with ext4 root- partition (journal activated)
> >
> >
> > After system running two weeks or so superblock from rootfs (ext4) at block 0 got overwritten with "trash data".
> > This is happening with like 20% of the embedded devices.
> >
> > hex comparision between faulty and good superblock starting at byte 1024:
> >
> > Before (good):
> >
> > 00000000 00 ee 02 00 00 b8 0b 00 00 96 00 00 ab a9 05 00 |................|
> > 00000010 c3 0c 02 00 00 00 00 00 02 00 00 00 02 00 00 00 |................|
> > 00000020 00 80 00 00 00 80 00 00 40 1f 00 00 9e 68 46 58 |........@....hFX|
> > 00000030 9e 68 46 58 2e 00 64 00 53 ef 01 00 01 00 00 00 |.hFX..d.S.......|
>
> <snip>
>
> >
> > After (corrupted):
> >
> > 00000000 00 00 00 00 a4 81 00 00 dd 00 00 00 24 8e 5d 54 |............$.]T|
> > 00000010 7e 8e 5d 54 18 a6 9f 41 00 00 00 00 00 00 01 00 |~.]T...A........|
> > 00000020 08 00 00 00 00 00 08 00 01 00 00 00 0a f3 01 00 |................|
> > 00000030 04 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................|

> 81a4? f30a? That looks like an inode block (i_mode 0x81a4) and an
> extent tree root (eh_magic 0xf30a), shifted up by 4 bytes before being
> written onto sector 0.

Thx for your replys! Wow, thats interesting. Here is another hex dump from another corrupted card (as mentioned it is a reproducible problem):

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000040  00 00 00 00 00 00 00 00  a4 81 00 00 85 06 00 00  |................|
00000050  25 e9 0b 56 f4 42 6a 52  59 7d 15 51 00 00 00 00  |%..V.BjRY}.Q....|
00000060  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000070  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  01 00 00 00 7f 82 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 9a df 58 e9  |..............X.|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000000d0  00 00 00 00 5c 57 7a 77  f4 42 6a 52 b0 29 af 96  |....\Wzw.BjR.)..|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000140  00 00 00 00 00 00 00 00  80 81 00 00 9c 02 00 00  |................|
00000150  74 db 84 57 f4 42 6a 52  d4 30 1a 52 00 00 00 00  |t..W.BjR.0.R....|
00000160  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000170  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  01 00 00 00 80 82 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 9b df 58 e9  |..............X.|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000001d0  00 00 00 00 78 93 71 17  f4 42 6a 52 b0 29 af 96  |....x.q..BjR.)..|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000240  00 00 00 00 00 00 00 00  a4 81 00 00 bc 00 00 00  |................|
00000250  22 4c 41 53 f4 42 6a 52  d4 30 1a 52 00 00 00 00  |"LAS.BjR.0.R....|
00000260  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000270  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  01 00 00 00 81 82 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 9c df 58 e9  |..............X.|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000002d0  00 00 00 00 bc 8b 80 a8  f4 42 6a 52 b0 29 af 96  |.........BjR.)..|
000002e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000340  00 00 00 00 00 00 00 00  a4 81 00 00 dc 13 02 00  |................|
00000350  7e dd 84 57 f4 42 6a 52  59 7d 15 51 00 00 00 00  |~..W.BjRY}.Q....|
00000360  00 00 01 00 10 01 00 00  00 00 08 00 01 00 00 00  |................|
00000370  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000380  22 00 00 00 82 82 00 00  00 00 00 00 00 00 00 00  |"...............|
00000390  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000003a0  00 00 00 00 00 00 00 00  00 00 00 00 9d df 58 e9  |..............X.|
000003b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000003d0  00 00 00 00 d8 ed 4c 81  f4 42 6a 52 b0 29 af 96  |......L..BjR.)..|
000003e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000440  00 00 00 00 00 00 00 00  80 81 00 00 e3 00 00 00  |................|
00000450  74 db 84 57 f4 42 6a 52  d4 30 1a 52 00 00 00 00  |t..W.BjR.0.R....|
00000460  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000470  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000480  01 00 00 00 a4 82 00 00  00 00 00 00 00 00 00 00  |................|
00000490  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000004a0  00 00 00 00 00 00 00 00  00 00 00 00 9e df 58 e9  |..............X.|
000004b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000004c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000004d0  00 00 00 00 78 3f 2b 1b  f4 42 6a 52 b0 29 af 96  |....x?+..BjR.)..|
000004e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000540  00 00 00 00 00 00 00 00  a4 81 00 00 de 1b 00 00  |................|
00000550  93 c5 fe 54 f4 42 6a 52  5f 31 1a 52 00 00 00 00  |...T.BjR_1.R....|
00000560  00 00 01 00 10 00 00 00  00 00 08 00 01 00 00 00  |................|
00000570  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000580  02 00 00 00 a6 82 00 00  00 00 00 00 00 00 00 00  |................|
00000590  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000005a0  00 00 00 00 00 00 00 00  00 00 00 00 9f df 58 e9  |..............X.|
000005b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000005c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000005d0  00 00 00 00 34 b9 b4 05  f4 42 6a 52 b0 29 af 96  |....4....BjR.)..|
000005e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000640  00 00 00 00 00 00 00 00  ed 41 00 00 00 10 00 00  |.........A......|
00000650  1d d0 bb 57 32 42 ad 53  32 42 ad 53 00 00 00 00  |...W2B.S2B.S....|
00000660  00 00 02 00 08 00 00 00  00 00 08 00 16 00 00 00  |................|
00000670  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000680  01 00 00 00 91 20 00 00  00 00 00 00 00 00 00 00  |..... ..........|
00000690  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000006a0  00 00 00 00 00 00 00 00  00 00 00 00 a0 df 58 e9  |..............X.|
000006b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000006c0  00 00 00 00 00 00 00 00  1c 00 00 00 10 ec 28 b8  |..............(.|
000006d0  10 ec 28 b8 e0 2b ac 8b  f4 42 6a 52 b0 29 af 96  |..(..+...BjR.)..|
000006e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000740  00 00 00 00 00 00 00 00  ed 81 00 00 00 01 00 00  |................|
00000750  f3 d0 bb 57 f4 42 6a 52  59 b3 46 51 00 00 00 00  |...W.BjRY.FQ....|
00000760  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000770  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000780  01 00 00 00 a5 82 00 00  00 00 00 00 00 00 00 00  |................|
00000790  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000007a0  00 00 00 00 00 00 00 00  00 00 00 00 a1 df 58 e9  |..............X.|
000007b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000007c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000007d0  00 00 00 00 68 68 e2 0c  f4 42 6a 52 b0 29 af 96  |....hh...BjR.)..|
000007e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000840  00 00 00 00 00 00 00 00  ed 81 00 00 79 02 00 00  |............y...|
00000850  1d d0 bb 57 f4 42 6a 52  53 21 35 51 00 00 00 00  |...W.BjRS!5Q....|
00000860  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000870  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000880  01 00 00 00 a8 82 00 00  00 00 00 00 00 00 00 00  |................|
00000890  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000008a0  00 00 00 00 00 00 00 00  00 00 00 00 a2 df 58 e9  |..............X.|
000008b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000008c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000008d0  00 00 00 00 e4 ad 42 91  f4 42 6a 52 b0 29 af 96  |......B..BjR.)..|
000008e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000940  00 00 00 00 00 00 00 00  ed 81 00 00 59 00 00 00  |............Y...|
00000950  19 2e b9 57 f4 42 6a 52  9d 10 b5 4f 00 00 00 00  |...W.BjR...O....|
00000960  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000970  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000980  01 00 00 00 a9 82 00 00  00 00 00 00 00 00 00 00  |................|
00000990  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000009a0  00 00 00 00 00 00 00 00  00 00 00 00 a3 df 58 e9  |..............X.|
000009b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000009c0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
000009d0  00 00 00 00 68 63 ea 99  f4 42 6a 52 b0 29 af 96  |....hc...BjR.)..|
000009e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000a40  00 00 00 00 00 00 00 00  ed 81 00 00 63 01 00 00  |............c...|
00000a50  f3 d0 bb 57 f4 42 6a 52  bb df d5 4f 00 00 00 00  |...W.BjR...O....|
00000a60  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000a70  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000a80  01 00 00 00 aa 82 00 00  00 00 00 00 00 00 00 00  |................|
00000a90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 a4 df 58 e9  |..............X.|
00000ab0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000ac0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
00000ad0  00 00 00 00 68 bc 28 09  f4 42 6a 52 b0 29 af 96  |....h.(..BjR.)..|
00000ae0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000b40  00 00 00 00 00 00 00 00  ed 81 00 00 89 3a 00 00  |.............:..|
00000b50  1d d0 bb 57 f4 42 6a 52  77 15 b0 51 00 00 00 00  |...W.BjRw..Q....|
00000b60  00 00 01 00 20 00 00 00  00 00 08 00 01 00 00 00  |.... ...........|
00000b70  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000b80  04 00 00 00 ac 82 00 00  00 00 00 00 00 00 00 00  |................|
00000b90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000ba0  00 00 00 00 00 00 00 00  00 00 00 00 a5 df 58 e9  |..............X.|
00000bb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000bc0  00 00 00 00 00 00 00 00  1c 00 00 00 b0 29 af 96  |.............)..|
00000bd0  00 00 00 00 ec b1 6f 9c  f4 42 6a 52 b0 29 af 96  |......o..BjR.)..|
00000be0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000c40  00 00 00 00 00 00 00 00  a4 81 00 00 66 00 00 00  |............f...|
00000c50  f8 87 98 45 f4 42 6a 52  3e 6f f3 4f 00 00 00 00  |...E.BjR>o.O....|
00000c60  00 00 01 00 08 00 00 00  00 00 08 00 01 00 00 00  |................|
00000c70  0a f3 01 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
00000c80  01 00 00 00 ab 82 00 00  00 00 00 00 00 00 00 00  |................|
00000c90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000ca0  00 00 00 00 00 00 00 00  00 00 00 00 a6 df 58 e9  |..............X.|
00000cb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000cc0  00 00 00 00 00 00 00 00  1c 00 00 00 ac 34 ec 96  |.............4..|
00000cd0  00 00 00 00 a8 cb 1c ad  f4 42 6a 52 ac 34 ec 96  |.........BjR.4..|
00000ce0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000d40  00 00 00 00 00 00 00 00  ff a1 00 00 19 00 00 00  |................|
00000d50  73 75 16 56 f7 42 6a 52  70 31 1a 52 00 00 00 00  |su.V.BjRp1.R....|
00000d60  00 00 01 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000d70  2e 2e 2f 69 6e 69 74 2e  64 2f 62 6f 6f 74 5f 73  |../init.d/boot_s|
00000d80  63 72 69 70 74 73 2e 73  68 00 00 00 00 00 00 00  |cripts.sh.......|
00000d90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000da0  00 00 00 00 00 00 00 00  00 00 00 00 c8 32 59 e9  |.............2Y.|
00000db0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000dc0  00 00 00 00 00 00 00 00  1c 00 00 00 30 48 96 d8  |............0H..|
00000dd0  00 00 00 00 68 a6 5a 83  f7 42 6a 52 30 48 96 d8  |....h.Z..BjR0H..|
00000de0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000e40  00 00 00 00 00 00 00 00  ff a1 00 00 0e 00 00 00  |................|
00000e50  73 75 16 56 cb a3 0a 56  cb a3 0a 56 00 00 00 00  |su.V...V...V....|
00000e60  00 00 01 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000e70  2e 2e 2f 69 6e 69 74 2e  64 2f 63 72 6f 6e 00 00  |../init.d/cron..|
00000e80  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ea0  00 00 00 00 00 00 00 00  00 00 00 00 31 92 30 60  |............1.0`|
00000eb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000ec0  00 00 00 00 00 00 00 00  1c 00 00 00 e0 32 a8 1b  |.............2..|
00000ed0  e0 32 a8 1b 78 ae b4 99  cb a3 0a 56 e0 32 a8 1b  |.2..x......V.2..|
00000ee0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000f40  00 00 00 00 00 00 00 00  ff a1 00 00 0e 00 00 00  |................|
00000f50  73 75 16 56 cb a3 0a 56  cb a3 0a 56 00 00 00 00  |su.V...V...V....|
00000f60  00 00 01 00 00 00 00 00  00 00 00 00 01 00 00 00  |................|
00000f70  2e 2e 2f 69 6e 69 74 2e  64 2f 63 72 6f 6e 00 00  |../init.d/cron..|
00000f80  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000fa0  00 00 00 00 00 00 00 00  00 00 00 00 32 92 30 60  |............2.0`|
00000fb0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000fc0  00 00 00 00 00 00 00 00  1c 00 00 00 e4 08 85 1d  |................|
00000fd0  e4 08 85 1d 8c ba 3b bb  cb a3 0a 56 e4 08 85 1d  |......;....V....|
00000fe0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

> <snip>
>
> > - How is it possible, that even the magic number (and everything else) got overwritten?
> > - Why could it ever be overwritten?
>
> I don't think anyone here can tell you what happened, it is almost certainly not
> an ext4 bug. Could be a driver bug, or an admin running a stray "dd" command,
> or some other utility gone astray, or ... anything, really.

I can rule out that it was the dd command. Eric, do you think it could also be a problem about the SD-card and wear levelling (just a guess)?


> Bad/malicious SDHC card would be my guess? (Are you doing power fail testing?)

We did have the problem only with the same type of SD-cards, which on the other hand we used most frequently also. 
Don´t know what you mean by power fail testing: We tested power fail with our devices by frequently switching off devices (without any impact).
What we observed that the devices were dying "slowely", so it took some time till they werent accesible anymore.

Thx again!