From: "Don Bowman" Subject: ext4 corruption on md [7x1TB in RAID5] Date: Sat, 9 May 2009 13:14:56 -0400 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 8BIT To: Return-path: Received: from Mail1.sandvine.com ([64.7.137.134]:26991 "EHLO mail2.sandvine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752121AbZEIRbJ convert rfc822-to-8bit (ORCPT ); Sat, 9 May 2009 13:31:09 -0400 Content-class: urn:content-classes:message Sender: linux-ext4-owner@vger.kernel.org List-ID: I cleanly shut down my system, and when it came back up, this was reported: * Checking file systems... Sat May 9 13:00:53 2009 fsck 1.41.4 (27-Jan-2009) fsck.ext4: Unable to resolve 'UUID=52e18bf7-cbe9-4f58-824c-5c943dd074de'^M fsck died with exit status 8 For some reason it thinks it is ext2: root@server:/var/log/fsck# fsck -n /dev/md0 | head -5 fsck 1.41.4 (27-Jan-2009) e2fsck 1.41.4 (27-Jan-2009) fsck.ext2: Group descriptors look bad... trying backup blocks... Inode table for group 0 is not in group. (block 3251545658) WARNING: SEVERE DATA LOSS POSSIBLE. Relocate? No Upon investigation, i'm not sure what to do. I really don't want to lose the data on this disk as it will take ages to rebuild (like weeks!). The system has been in operation for about 2-3months. The data on the disk is mostly video + some virtual machines. Any suggestions for repairing it? I have not done any repairs so far. Should i just Run fsck.ext4? Is there something i should try? root@server:/var/log/fsck# uname -a Linux server 2.6.28-12-server #43-Ubuntu SMP Fri May 1 20:22:39 UTC 2009 x86_64 GNU/Linux root@server:/var/log/fsck# fsck.ext4 -n /dev/md0 | head -5 e2fsck 1.41.4 (27-Jan-2009) fsck.ext4: Group descriptors look bad... trying backup blocks... Inode table for group 0 is not in group. (block 3251545658) WARNING: SEVERE DATA LOSS POSSIBLE. Relocate? No root@server:~# mdadm --detail /dev/md0 /dev/md0: Version : 00.90 Creation Time : Sun Mar 15 12:15:20 2009 Raid Level : raid5 Array Size : 5860574976 (5589.08 GiB 6001.23 GB) Used Dev Size : 976762496 (931.51 GiB 1000.20 GB) Raid Devices : 7 Total Devices : 7 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat May 9 12:42:47 2009 State : clean Active Devices : 7 Working Devices : 7 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K UUID : c589dbe9:8dfff933:01f9e43d:ac30fbff (local to host server) Events : 0.305524 Number Major Minor RaidDevice State 0 8 0 0 active sync /dev/sda 1 8 16 1 active sync /dev/sdb 2 8 32 2 active sync /dev/sdc 3 8 64 3 active sync /dev/sde 4 8 80 4 active sync /dev/sdf 5 8 96 5 active sync /dev/sdg 6 8 112 6 active sync /dev/sdh root@server:~# root@server:/var/log/fsck# dumpe2fs /dev/md0 |head -60 dumpe2fs 1.41.4 (27-Jan-2009) ext2fs_read_bb_inode: Invalid argument Filesystem volume name: Last mounted on: Filesystem UUID: 52e18bf7-cbe9-4f58-824c-5c943dd074de Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: not clean with errors Errors behavior: Continue Filesystem OS type: Linux Inode count: 366288896 Block count: 1465143744 Reserved block count: 73257187 Free blocks: 1041313557 Free inodes: 366151494 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 674 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Sun Mar 15 12:17:10 2009 Last mount time: Wed Apr 22 19:04:50 2009 Last write time: Sat May 9 12:31:43 2009 Mount count: 9 Maximum mount count: 29 Last checked: Sun Mar 15 12:17:10 2009 Check interval: 15552000 (6 months) Next check after: Fri Sep 11 12:17:10 2009 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Default directory hash: half_md4 Directory Hash Seed: 49785c19-388f-444c-977a-886c147a8ae6 Journal backup: inode blocks Group 0: (Blocks 0-32767) Checksum 0x70e6, unused inodes 34168 Primary superblock at 0, Group descriptors at 1-350 Reserved GDT blocks at 351-1024 Block bitmap at 98318778 (+98318778), Inode bitmap at 837719297 (+837719297) Inode table at 3251545658-3251546169 (+3251545658) 58662 free blocks, 33159 free inodes, 56553 directories, 34168 unused inodes Group 1: (Blocks 32768-65535) [INODE_UNINIT, ITABLE_ZEROED] Checksum 0x6275, unused inodes 29973 Backup superblock at 32768, Group descriptors at 32769-33118 Reserved GDT blocks at 33119-33792 Block bitmap at 1123273429 (+1123240661), Inode bitmap at 3031071618 (+3031038850) Inode table at 3149300410-3149300921 (+3149267642) 1881 free blocks, 46059 free inodes, 25281 directories, 29973 unused inodes Group 2: (Blocks 65536-98303) [INODE_UNINIT, ITABLE_ZEROED] Checksum 0x1a0b, unused inodes 51590 Block bitmap at 1509085182 (+1509019646), Inode bitmap at 2884607537 (+2884542001) Inode table at 1940584034-1940584545 (+1940518498) 3770 free blocks, 40683 free inodes, 43709 directories, 51590 unused inodes Group 3: (Blocks 98304-131071) [ITABLE_ZEROED] Checksum 0xc3ec, unused inodes 47423 Backup superblock at 98304, Group descriptors at 98305-98654 Reserved GDT blocks at 98655-99328 Block bitmap at 383505502 (+383407198), Inode bitmap at 1784600367 (+1784502063) Inode table at 798930664-798931175 (+798832360) 63118 free blocks, 7143 free inodes, 19773 directories, 47423 unused inodes root@server:/var/log/fsck# dmesg |egrep -i "sd|md|raid|fsck" [ 0.000000] AMD AuthenticAMD [ 0.000000] RAMDISK: 37778000 - 37fef9b2 [ 0.000000] ACPI: RSDP 000FE020, 0014 (r0 INTEL ) [ 0.000000] ACPI: RSDT CFEFD038, 004C (r1 INTEL D975XBX2 AE8 1000013) [ 0.000000] ACPI: DSDT CFEF8000, 3F11 (r1 INTEL D975XBX2 AE8 MSFT 1000013) [ 0.000000] ACPI: SSDT CFEF2000, 01BC (r1 INTEL CpuPm AE8 MSFT 1000013) [ 0.000000] ACPI: SSDT CFEF1000, 0175 (r1 INTEL Cpu0Ist AE8 MSFT 1000013) [ 0.000000] ACPI: SSDT CFEF0000, 0175 (r1 INTEL Cpu1Ist AE8 MSFT 1000013) [ 0.000000] ACPI: SSDT CFEAB000, 0175 (r1 INTEL Cpu2Ist AE8 MSFT 1000013) [ 0.000000] ACPI: SSDT CFEAA000, 0175 (r1 INTEL Cpu3Ist AE8 MSFT 1000013) [ 0.000000] #3 [0037778000 - 0037fef9b2] RAMDISK ==> [0037778000 - 0037fef9b2] [ 0.000000] [ffffe20000000000-ffffe200043fffff] PMD -> [ffff880028200000-ffff88002c5fffff] on node 0 [ 0.013732] ACPI: Checking initramfs for custom DSDT [ 0.920745] ACPI: EC: Look up EC in DSDT [ 1.835081] Fixed MDIO Bus: probed [ 1.835147] Driver 'sd' needs updating - please use bus_type methods [ 1.835891] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x40b0 irq 14 [ 1.835893] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x40b8 irq 15 [ 2.056689] ata3: SATA max UDMA/133 cmd 0x40c8 ctl 0x40e4 bmdma 0x40a0 irq 19 [ 2.056691] ata4: SATA max UDMA/133 cmd 0x40c0 ctl 0x40e0 bmdma 0x40a8 irq 19 [ 2.250605] ata3.00: ATA-8: ST31000340AS, SD1A, max UDMA/133 [ 2.510628] ata4.00: ATA-8: ST31000340AS, SD1A, max UDMA/133 [ 2.591311] scsi 2:0:0:0: Direct-Access ATA ST31000340AS SD1A PQ: 0 ANSI: 5 [ 2.591402] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 2.591414] sd 2:0:0:0: [sda] Write Protect is off [ 2.591416] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 2.591435] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.591478] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 2.591489] sd 2:0:0:0: [sda] Write Protect is off [ 2.591490] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 2.591508] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.591511] sda: unknown partition table [ 2.600717] sd 2:0:0:0: [sda] Attached SCSI disk [ 2.600745] sd 2:0:0:0: Attached scsi generic sg1 type 0 [ 2.600855] sd 2:0:1:0: [sdb] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 2.600866] sd 2:0:1:0: [sdb] Write Protect is off [ 2.600868] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00 [ 2.600886] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.600924] sd 2:0:1:0: [sdb] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 2.600934] sd 2:0:1:0: [sdb] Write Protect is off [ 2.600936] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00 [ 2.600954] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.600957] sdb: unknown partition table [ 3.072070] sd 2:0:1:0: [sdb] Attached SCSI disk [ 3.072115] sd 2:0:1:0: Attached scsi generic sg2 type 0 [ 3.072178] scsi 3:0:0:0: Direct-Access ATA ST31000340AS SD1A PQ: 0 ANSI: 5 [ 3.072237] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 3.072248] sd 3:0:0:0: [sdc] Write Protect is off [ 3.072249] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 3.072268] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.072303] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 3.072313] sd 3:0:0:0: [sdc] Write Protect is off [ 3.072315] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 3.072333] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.072336] sdc: unknown partition table [ 3.079128] sd 3:0:0:0: [sdc] Attached SCSI disk [ 3.079156] sd 3:0:0:0: Attached scsi generic sg3 type 0 [ 3.079257] sd 3:0:1:0: [sdd] 293046768 512-byte hardware sectors: (150 GB/139 GiB) [ 3.079268] sd 3:0:1:0: [sdd] Write Protect is off [ 3.079269] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00 [ 3.079287] sd 3:0:1:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.079322] sd 3:0:1:0: [sdd] 293046768 512-byte hardware sectors: (150 GB/139 GiB) [ 3.079332] sd 3:0:1:0: [sdd] Write Protect is off [ 3.079334] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00 [ 3.079352] sd 3:0:1:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.079354] sdd: sdd1 sdd2 < sdd5 > [ 3.101054] sd 3:0:1:0: [sdd] Attached SCSI disk [ 3.101107] sd 3:0:1:0: Attached scsi generic sg4 type 0 [ 3.470676] ata5.00: ATA-8: ST31000340AS, SD1A, max UDMA/133 [ 3.880672] ata6.00: ATA-8: ST31000340AS, SD1A, max UDMA/133 [ 4.290698] ata7.00: ATA-8: ST31000340AS, SD1A, max UDMA/133 [ 4.700696] ata8.00: ATA-8: ST31000340AS, SD1A, max UDMA/133 [ 4.740750] scsi 4:0:0:0: Direct-Access ATA ST31000340AS SD1A PQ: 0 ANSI: 5 [ 4.740831] sd 4:0:0:0: [sde] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.740842] sd 4:0:0:0: [sde] Write Protect is off [ 4.740844] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00 [ 4.740862] sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.740898] sd 4:0:0:0: [sde] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.740909] sd 4:0:0:0: [sde] Write Protect is off [ 4.740910] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00 [ 4.740928] sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.740930] sde: unknown partition table [ 4.749416] sd 4:0:0:0: [sde] Attached SCSI disk [ 4.749443] sd 4:0:0:0: Attached scsi generic sg5 type 0 [ 4.749488] scsi 5:0:0:0: Direct-Access ATA ST31000340AS SD1A PQ: 0 ANSI: 5 [ 4.749546] sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.749557] sd 5:0:0:0: [sdf] Write Protect is off [ 4.749559] sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 [ 4.749576] sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.749613] sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.749623] sd 5:0:0:0: [sdf] Write Protect is off [ 4.749624] sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 [ 4.749642] sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.749645] sdf: unknown partition table [ 4.759788] sd 5:0:0:0: [sdf] Attached SCSI disk [ 4.759815] sd 5:0:0:0: Attached scsi generic sg6 type 0 [ 4.759859] scsi 6:0:0:0: Direct-Access ATA ST31000340AS SD1A PQ: 0 ANSI: 5 [ 4.759919] sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.759930] sd 6:0:0:0: [sdg] Write Protect is off [ 4.759932] sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00 [ 4.759950] sd 6:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.759986] sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.759997] sd 6:0:0:0: [sdg] Write Protect is off [ 4.759998] sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00 [ 4.760022] sd 6:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.760027] sdg: unknown partition table [ 4.771408] sd 6:0:0:0: [sdg] Attached SCSI disk [ 4.771439] sd 6:0:0:0: Attached scsi generic sg7 type 0 [ 4.771483] scsi 7:0:0:0: Direct-Access ATA ST31000340AS SD1A PQ: 0 ANSI: 5 [ 4.771540] sd 7:0:0:0: [sdh] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.771551] sd 7:0:0:0: [sdh] Write Protect is off [ 4.771552] sd 7:0:0:0: [sdh] Mode Sense: 00 3a 00 00 [ 4.771570] sd 7:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.771608] sd 7:0:0:0: [sdh] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) [ 4.771618] sd 7:0:0:0: [sdh] Write Protect is off [ 4.771620] sd 7:0:0:0: [sdh] Mode Sense: 00 3a 00 00 [ 4.771637] sd 7:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 4.771640] sdh: unknown partition table [ 4.781982] sd 7:0:0:0: [sdh] Attached SCSI disk [ 4.782012] sd 7:0:0:0: Attached scsi generic sg8 type 0 [ 4.942482] block sdd5: hash matches [ 5.048354] md: linear personality registered for level -1 [ 5.050051] md: multipath personality registered for level -4 [ 5.051538] md: raid0 personality registered for level 0 [ 5.053697] md: raid1 personality registered for level 1 [ 5.271272] raid6: int64x1 1922 MB/s [ 5.441269] raid6: int64x2 2639 MB/s [ 5.611280] raid6: int64x4 2025 MB/s [ 5.781295] raid6: int64x8 1789 MB/s [ 5.951290] raid6: sse2x1 3078 MB/s [ 6.121297] raid6: sse2x2 3651 MB/s [ 6.291275] raid6: sse2x4 7193 MB/s [ 6.291281] raid6: using algorithm sse2x4 (7193 MB/s) [ 6.291283] md: raid6 personality registered for level 6 [ 6.291285] md: raid5 personality registered for level 5 [ 6.291286] md: raid4 personality registered for level 4 [ 6.296306] md: raid10 personality registered for level 10 [ 6.494887] md: bind [ 6.555824] md: bind [ 6.622315] md: bind [ 6.703948] md: bind [ 6.780815] md: bind [ 7.023507] md: bind [ 7.223400] md: bind [ 7.226582] raid5: device sdb operational as raid disk 1 [ 7.226584] raid5: device sdc operational as raid disk 2 [ 7.226586] raid5: device sdg operational as raid disk 5 [ 7.226587] raid5: device sda operational as raid disk 0 [ 7.226588] raid5: device sdh operational as raid disk 6 [ 7.226589] raid5: device sdf operational as raid disk 4 [ 7.226591] raid5: device sde operational as raid disk 3 [ 7.227201] raid5: allocated 7450kB for md0 [ 7.227203] raid5: raid level 5 set md0 active with 7 out of 7 devices, algorithm 2 [ 7.227205] RAID5 conf printout: [ 7.227207] disk 0, o:1, dev:sda [ 7.227208] disk 1, o:1, dev:sdb [ 7.227209] disk 2, o:1, dev:sdc [ 7.227210] disk 3, o:1, dev:sde [ 7.227212] disk 4, o:1, dev:sdf [ 7.227213] disk 5, o:1, dev:sdg [ 7.227214] disk 6, o:1, dev:sdh [ 7.228333] md0: unknown partition table [ 12.709829] Adding 5992204k swap on /dev/sdd5. Priority:-1 extents:1 across:5992204k [ 13.245731] EXT3 FS on sdd1, internal journal [ 317.618298] type=1505 audit(1241888758.168:7): operation="profile_load" name="/usr/sbin/cupsd" name2="default" pid=2550 [ 318.016442] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 322.767731] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 322.774119] NFSD: starting 90-second grace period