Maybe this is deletion related since I was creating and destroying a
bunch of file with rsync. I don't know. I'm redoing the rsync with
checksums to see if the data is still in tact. Seems like some bits of
this got corrupted, but I can't tell if it's disk or file system
related.
bill
------------------------------------
Feb 22 19:23:31 finfin kernel: [ 2.819633] sdb1
Feb 22 19:27:40 finfin kernel: [ 263.857108] sdb: sdb1
Feb 22 20:03:18 finfin kernel: [ 2402.182269] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 03:49:40 finfin kernel: [203184.800029] EXT4-fs (sdb1):
warning: mounting fs with errors, running e2fsck is recommended
Feb 25 03:49:41 finfin kernel: [203184.980730] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 04:41:26 finfin kernel: [206290.181230] JBD: Spotted dirty
metadata buffer (dev = sdb1, blocknr = 0). There's a risk of
filesystem corruption in case of system crash.
Feb 25 05:01:39 finfin kernel: [ 2.405998] sdb: sdb1
Feb 25 05:02:07 finfin kernel: [ 82.487625] EXT4-fs (sdb1): warning:
mounting fs with errors, running e2fsck is recommended
Feb 25 05:02:07 finfin kernel: [ 82.657297] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 05:09:40 finfin kernel: [ 2.408385] sdb: sdb1
Feb 25 05:14:17 finfin kernel: [ 2.438605] sdb: sdb1
Feb 25 05:16:21 finfin kernel: [ 174.548463] EXT4-fs (sdb1): warning:
mounting fs with errors, running e2fsck is recommended
Feb 25 05:16:21 finfin kernel: [ 174.728174] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 15:18:20 finfin kernel: [36293.209497] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242283) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209594] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242282) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209668] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242281) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209739] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242280) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209745] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (8)
Feb 25 15:18:20 finfin kernel: [36293.209818] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242279) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209824] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (5)
Feb 25 15:18:20 finfin kernel: [36293.209892] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242278) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209898] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (3)
Feb 25 15:18:20 finfin kernel: [36293.226488] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (125)
Feb 25 15:18:45 finfin kernel: [36317.996660] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40247537) - no data block
Feb 25 15:18:45 finfin kernel: [36317.996671] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (24906)
Feb 25 15:19:38 finfin kernel: [36371.180386] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40239122) - no `.' or `..'
Feb 25 15:19:38 finfin kernel: [36371.232941] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (3)
Feb 25 15:19:38 finfin kernel: [36371.343549] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (5)
Feb 25 15:19:38 finfin kernel: [36371.397308] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40239106) - no `.' or `..'
Feb 25 18:13:27 finfin kernel: [46799.800244] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 21:51:47 finfin kernel: [59900.021575] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
On Feb 26, 2011, at 5:16 AM, Bill Huey (hui) wrote:
> Maybe this is deletion related since I was creating and destroying a
> bunch of file with rsync. I don't know. I'm redoing the rsync with
> checksums to see if the data is still in tact. Seems like some bits of
> this got corrupted, but I can't tell if it's disk or file system
> related.
I'd run e2fsck before you do anything else. Some of the error
messages listed below, especially the ones about bad link
counts on empty directories, smells very much like the inode
table got corrupted.
-- Ted
>
> bill
> ------------------------------------
>
> Feb 22 19:23:31 finfin kernel: [ 2.819633] sdb1
> Feb 22 19:27:40 finfin kernel: [ 263.857108] sdb: sdb1
> Feb 22 20:03:18 finfin kernel: [ 2402.182269] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 25 03:49:40 finfin kernel: [203184.800029] EXT4-fs (sdb1):
> warning: mounting fs with errors, running e2fsck is recommended
> Feb 25 03:49:41 finfin kernel: [203184.980730] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 25 04:41:26 finfin kernel: [206290.181230] JBD: Spotted dirty
> metadata buffer (dev = sdb1, blocknr = 0). There's a risk of
> filesystem corruption in case of system crash.
> Feb 25 05:01:39 finfin kernel: [ 2.405998] sdb: sdb1
> Feb 25 05:02:07 finfin kernel: [ 82.487625] EXT4-fs (sdb1): warning:
> mounting fs with errors, running e2fsck is recommended
> Feb 25 05:02:07 finfin kernel: [ 82.657297] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 25 05:09:40 finfin kernel: [ 2.408385] sdb: sdb1
> Feb 25 05:14:17 finfin kernel: [ 2.438605] sdb: sdb1
> Feb 25 05:16:21 finfin kernel: [ 174.548463] EXT4-fs (sdb1): warning:
> mounting fs with errors, running e2fsck is recommended
> Feb 25 05:16:21 finfin kernel: [ 174.728174] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 25 15:18:20 finfin kernel: [36293.209497] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40242283) - no `.' or `..'
> Feb 25 15:18:20 finfin kernel: [36293.209594] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40242282) - no `.' or `..'
> Feb 25 15:18:20 finfin kernel: [36293.209668] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40242281) - no `.' or `..'
> Feb 25 15:18:20 finfin kernel: [36293.209739] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40242280) - no `.' or `..'
> Feb 25 15:18:20 finfin kernel: [36293.209745] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (8)
> Feb 25 15:18:20 finfin kernel: [36293.209818] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40242279) - no `.' or `..'
> Feb 25 15:18:20 finfin kernel: [36293.209824] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (5)
> Feb 25 15:18:20 finfin kernel: [36293.209892] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40242278) - no `.' or `..'
> Feb 25 15:18:20 finfin kernel: [36293.209898] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (3)
> Feb 25 15:18:20 finfin kernel: [36293.226488] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (125)
> Feb 25 15:18:45 finfin kernel: [36317.996660] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40247537) - no data block
> Feb 25 15:18:45 finfin kernel: [36317.996671] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (24906)
> Feb 25 15:19:38 finfin kernel: [36371.180386] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40239122) - no `.' or `..'
> Feb 25 15:19:38 finfin kernel: [36371.232941] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (3)
> Feb 25 15:19:38 finfin kernel: [36371.343549] EXT4-fs warning (device
> sdb1): ext4_rmdir: empty directory has too many links (5)
> Feb 25 15:19:38 finfin kernel: [36371.397308] EXT4-fs warning (device
> sdb1): empty_dir: bad directory (dir #40239106) - no `.' or `..'
> Feb 25 18:13:27 finfin kernel: [46799.800244] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 25 21:51:47 finfin kernel: [59900.021575] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Theodore,
I did run fsck.ext4 on the file system. It cleared a bunch of errors
and but it's still showing various problems.
Another log of the file system. USB->SATA problems ?
---------------------------------
Feb 22 19:23:31 finfin kernel: [ 2.402463] sd 6:0:0:0: [sdb]
3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Feb 22 19:23:31 finfin kernel: [ 2.403585] sd 6:0:0:0: [sdb] Write
Protect is off
Feb 22 19:23:31 finfin kernel: [ 2.405920] sdb:
Feb 22 19:23:31 finfin kernel: [ 2.819633] sdb1
Feb 22 19:23:31 finfin kernel: [ 2.821592] sd 6:0:0:0: [sdb]
Attached SCSI disk
Feb 22 19:27:40 finfin kernel: [ 263.857108] sdb: sdb1
Feb 22 20:03:18 finfin kernel: [ 2402.182269] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 03:49:40 finfin kernel: [203184.800029] EXT4-fs (sdb1):
warning: mounting fs with errors, running e2fsck is recommended
Feb 25 03:49:41 finfin kernel: [203184.980730] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 04:41:26 finfin kernel: [206290.181230] JBD: Spotted dirty
metadata buffer (dev = sdb1, blocknr = 0). There's a risk of
filesystem corruption in case of system crash.
Feb 25 05:01:39 finfin kernel: [ 2.402538] sd 6:0:0:0: [sdb]
3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Feb 25 05:01:39 finfin kernel: [ 2.403659] sd 6:0:0:0: [sdb] Write
Protect is off
Feb 25 05:01:39 finfin kernel: [ 2.405998] sdb: sdb1
Feb 25 05:01:39 finfin kernel: [ 2.424879] sd 6:0:0:0: [sdb]
Attached SCSI disk
Feb 25 05:02:07 finfin kernel: [ 82.487625] EXT4-fs (sdb1): warning:
mounting fs with errors, running e2fsck is recommended
Feb 25 05:02:07 finfin kernel: [ 82.657297] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 05:09:40 finfin kernel: [ 2.404933] sd 6:0:0:0: [sdb]
3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Feb 25 05:09:40 finfin kernel: [ 2.406043] sd 6:0:0:0: [sdb] Write
Protect is off
Feb 25 05:09:40 finfin kernel: [ 2.408385] sdb: sdb1
Feb 25 05:09:40 finfin kernel: [ 2.411134] sd 6:0:0:0: [sdb]
Attached SCSI disk
Feb 25 05:14:17 finfin kernel: [ 2.435019] sd 6:0:0:0: [sdb]
3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Feb 25 05:14:17 finfin kernel: [ 2.436139] sd 6:0:0:0: [sdb] Write
Protect is off
Feb 25 05:14:17 finfin kernel: [ 2.438605] sdb: sdb1
Feb 25 05:14:17 finfin kernel: [ 2.441353] sd 6:0:0:0: [sdb]
Attached SCSI disk
Feb 25 05:16:21 finfin kernel: [ 174.548463] EXT4-fs (sdb1): warning:
mounting fs with errors, running e2fsck is recommended
Feb 25 05:16:21 finfin kernel: [ 174.728174] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 15:18:20 finfin kernel: [36293.209497] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242283) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209594] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242282) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209668] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242281) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209739] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242280) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209745] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (8)
Feb 25 15:18:20 finfin kernel: [36293.209818] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242279) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209824] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (5)
Feb 25 15:18:20 finfin kernel: [36293.209892] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40242278) - no `.' or `..'
Feb 25 15:18:20 finfin kernel: [36293.209898] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (3)
Feb 25 15:18:20 finfin kernel: [36293.226488] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (125)
Feb 25 15:18:45 finfin kernel: [36317.996660] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40247537) - no data block
Feb 25 15:18:45 finfin kernel: [36317.996671] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (24906)
Feb 25 15:19:38 finfin kernel: [36371.180386] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40239122) - no `.' or `..'
Feb 25 15:19:38 finfin kernel: [36371.232941] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (3)
Feb 25 15:19:38 finfin kernel: [36371.343549] EXT4-fs warning (device
sdb1): ext4_rmdir: empty directory has too many links (5)
Feb 25 15:19:38 finfin kernel: [36371.397308] EXT4-fs warning (device
sdb1): empty_dir: bad directory (dir #40239106) - no `.' or `..'
Feb 25 18:13:27 finfin kernel: [46799.800244] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 25 21:51:47 finfin kernel: [59900.021575] EXT4-fs (sdb1): mounted
filesystem with ordered data mode. Opts: (null)
Feb 26 02:19:27 finfin kernel: [75959.826438] sd 6:0:0:0: [sdb]
Unhandled error code
Feb 26 02:19:27 finfin kernel: [75959.826444] sd 6:0:0:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Feb 26 02:19:27 finfin kernel: [75959.826450] sd 6:0:0:0: [sdb] CDB:
Read(10): 28 00 ac 44 00 3f 00 00 08 00
Just to clarify, it did clear the problems, but the continued use of
the file system resulted in more corruption. It's no longer usable. I
have to remake the file system and copy the data again.
bill
On 2/26/11 5:13 AM, Bill Huey (hui) wrote:
> Theodore,
>
> I did run fsck.ext4 on the file system. It cleared a bunch of errors
> and but it's still showing various problems.
>
> Another log of the file system. USB->SATA problems ?
> ---------------------------------
>
> Feb 22 19:23:31 finfin kernel: [ 2.402463] sd 6:0:0:0: [sdb]
> 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
> Feb 22 19:23:31 finfin kernel: [ 2.403585] sd 6:0:0:0: [sdb] Write
> Protect is off
> Feb 22 19:23:31 finfin kernel: [ 2.405920] sdb:
> Feb 22 19:23:31 finfin kernel: [ 2.819633] sdb1
> Feb 22 19:23:31 finfin kernel: [ 2.821592] sd 6:0:0:0: [sdb]
> Attached SCSI disk
> Feb 22 19:27:40 finfin kernel: [ 263.857108] sdb: sdb1
> Feb 22 20:03:18 finfin kernel: [ 2402.182269] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 25 03:49:40 finfin kernel: [203184.800029] EXT4-fs (sdb1):
> warning: mounting fs with errors, running e2fsck is recommended
this must not have been post-fsck, if it still is mounting with errors...?
...
> Feb 25 21:51:47 finfin kernel: [59900.021575] EXT4-fs (sdb1): mounted
> filesystem with ordered data mode. Opts: (null)
> Feb 26 02:19:27 finfin kernel: [75959.826438] sd 6:0:0:0: [sdb]
> Unhandled error code
> Feb 26 02:19:27 finfin kernel: [75959.826444] sd 6:0:0:0: [sdb]
> Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> Feb 26 02:19:27 finfin kernel: [75959.826450] sd 6:0:0:0: [sdb] CDB:
> Read(10): 28 00 ac 44 00 3f 00 00 08 00
and this does indeed look like storage problems, ext4 problems are
probably secondary.
-Eric
On Sat, Feb 26, 2011 at 03:16:38AM -0800, Bill Huey (hui) wrote:
> Just to clarify, it did clear the problems, but the continued use of
> the file system resulted in more corruption. It's no longer usable. I
> have to remake the file system and copy the data again.
You didn't say what version of the kernel you were using, but there
hasn't been any ext4 bugs that have caused this much corruption this
quickly; it sure smells like a hardware problem....
- Ted
What ever is in the latest Ubuntu maverick, 2.6.35 for the kernel.
The only thing that makes it not looking like it was a storage problem
is this line that comes before the SATA error:
-----
Feb 25 04:41:26 finfin kernel: [206290.181230] JBD: Spotted dirty
metadata buffer (dev = sdb1, blocknr = 0). There's a risk of
filesystem corruption in case of system crash.
-----
bill
...unless those two lines are linked somehow via a timeout or something...
On Mon, Feb 28, 2011 at 12:18 PM, Bill Huey (hui) <[email protected]> wrote:
> What ever is in the latest Ubuntu maverick, 2.6.35 for the kernel.
>
> The only thing that makes it not looking like it was a storage problem
> is this line that comes before the SATA error:
>
> -----
> Feb 25 04:41:26 finfin kernel: [206290.181230] JBD: Spotted dirty
> metadata buffer (dev = sdb1, blocknr = 0). There's a risk of
> filesystem corruption in case of system crash.
> -----
On Mon, Feb 28, 2011 at 12:18:16PM -0800, Bill Huey (hui) wrote:
> What ever is in the latest Ubuntu maverick, 2.6.35 for the kernel.
>
> The only thing that makes it not looking like it was a storage problem
> is this line that comes before the SATA error:
>
> -----
> Feb 25 04:41:26 finfin kernel: [206290.181230] JBD: Spotted dirty
> metadata buffer (dev = sdb1, blocknr = 0). There's a risk of
> filesystem corruption in case of system crash.
> -----
There are a few places where we update the superblock bypassing the
journal layer. (For example, when we set the RO_COMPAT_LARGE_FILE
feature flag if it wasn't previously set). Those should be cleaned
up, but it's not related to the rest of the scary-looking corruption
which you saw. The worst that might happen is specific superblock
update might get lost (i.e., the RO_COMPAT_LARGE_FILE feature flag) on
a crash before we commit some other superblock change to the journal.
- Ted
Yeah, it was just a standard mkfs so I doubt that option was
specified. The important thing for me here was to at least let you
folks know about it so that you can determine if this is significant
or not.
Thanks
bill
On Mon, Feb 28, 2011 at 2:55 PM, Ted Ts'o <[email protected]> wrote:
> There are a few places where we update the superblock bypassing the
> journal layer. (For example, when we set the RO_COMPAT_LARGE_FILE
> feature flag if it wasn't previously set). Those should be cleaned
> up, but it's not related to the rest of the scary-looking corruption
> which you saw. The worst that might happen is specific superblock
> update might get lost (i.e., the RO_COMPAT_LARGE_FILE feature flag) on
> a crash before we commit some other superblock change to the journal.