From: "Darrick J. Wong" Subject: Re: A couple of ext4 crashes with inlinedata/bigalloc Date: Mon, 25 Aug 2014 12:09:32 -0700 Message-ID: <20140825190932.GD22645@birch.djwong.org> References: <20140311030604.GK9875@birch.djwong.org> <20140408115425.GA23781@gmail.com> <20140408184003.GB8888@birch.djwong.org> <20140409045641.GA8908@gmail.com> <20140409180449.GA9177@birch.djwong.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: linux-ext4 To: "Theodore Ts'o" , Zheng Liu Return-path: Received: from userp1040.oracle.com ([156.151.31.81]:42186 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754045AbaHYTJi (ORCPT ); Mon, 25 Aug 2014 15:09:38 -0400 Content-Disposition: inline In-Reply-To: <20140409180449.GA9177@birch.djwong.org> Sender: linux-ext4-owner@vger.kernel.org List-ID: On Wed, Apr 09, 2014 at 11:04:49AM -0700, Darrick J. Wong wrote: > On Wed, Apr 09, 2014 at 12:56:41PM +0800, Zheng Liu wrote: > > On Tue, Apr 08, 2014 at 11:40:03AM -0700, Darrick J. Wong wrote: > > > On Tue, Apr 08, 2014 at 07:54:25PM +0800, Zheng Liu wrote: > > > > On Mon, Mar 10, 2014 at 08:06:04PM -0700, Darrick J. Wong wrote: > > > > > Hi all, > > > > > > > > > > A couple of errors to complain about before I send out the e2fsprogs patchbomb. > > > > > :) > > > > > > > > > > Zheng: I've been running the metadata checksum test with inline_data set. > > > > > flat_dir_test is a stress test which copies /usr/share/doc into a filesystem > > > > > and then "enlarges" the directories by recursively renaming "$foo" to > > > > > "$foo.longer", with the results below. e2fsck complained that the directories > > > > > involved (4156, 4251) have multiple links to the subdir inode. I'm not sure > > > > > what this is all about; the only (circumstantial) evidence I have is that it > > > > > goes away if I don't turn on inline_data. > > > > > > > > Hi Darrick, > > > > > > > > I guess that you were talking about this bug at LSF/MM submmit this > > > > year, right? Thanks for reporting this bug. I am trying to take a > > > > closer look at it. It would be great if you could tell me where I can > > > > find the your test program ('flat_dir_test'). > > > > > > It's buried in the metadata checksum test suite, alas. The test copies an > > > arbitrary directory (/usr/share/doc) to the filesystem, then does a depth-first > > > traversal of the copied tree, renaming everything it finds from "oldname" to > > > "oldname.longer" to force the kernel to shuffle directory entries all over the > > > place, and likely causing spill-out from formerly inlinedata directories. > > > > I write a simple script to try to reproduce the problem. But, > > unfortunately, I failed. The kernel is based on ext4/dev branch, and > > e2fsprogs is based on e2fsprogs/next branch. I paste the script below > > and the output of 'dumpe2fs -h'. Please let me know if I miss > > something. > > It is necessary to umount and then mount the FS in between populating the FS > and renaming the files. flat_dir_test also renames *everything*, which is why > you need the "-depth" so that the renames don't break find: > > find $mnt/testdir -depth | while read f; do > mv $f $f.longer > done > > This doesn't affect the script's ability to reproduce errors; it just makes it > run faster. The errors I see look like this: > > [ 51.207952] EXT4-fs error (device vda): ext4_generic_delete_entry:2118: inode #4332: block 687: comm mv: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=33188, rec_len=1201, name_len=0 > [ 51.221472] EXT4-fs error (device vda) in ext4_delete_entry:2178: IO failure > [ 51.230300] EXT4-fs warning (device vda): ext4_rename:3151: Deleting old file (4332), 2, error=-5 > [ 51.512388] EXT4-fs error (device vda): ext4_generic_delete_entry:2118: inode #4427: block 693: comm mv: bad entry in directory: directory entry across range - offset=2056(2056), inode=1397064974, rec_len=34064, name_len=69 > [ 51.524525] EXT4-fs error (device vda) in ext4_delete_entry:2178: IO failure Welp, I figured out why this was happening -- if renaming within the same directory causes inline data conversion during a add-then-delete-dirent operation, the delete operation incorrectly tries to delete out of i_block (which is now an extent tree), causing the FS error. Patch soon. --D > > --D > > > > > Thanks, > > - Zheng > > > > > > #!/bin/bash > > > > dev='/dev/sda1' > > mnt='/mnt/sda1' > > e2fsprogs_base="$HOME/projects/ext4-dev/e2fsprogs" > > mkfs="$e2fsprogs_base/misc/mke2fs" > > fsck="$e2fsprogs_base/e2fsck/e2fsck" > > > > sudo umount $mnt > > sudo $mkfs -t ext4 -O inline_data,metadata_csum,64bit $dev > > sudo mount -t ext4 $dev $mnt > > sudo chown wenqing:wenqing $mnt > > > > cd $mnt > > mkdir testdir > > cp -rf /usr/share/doc/* testdir/ > > > > function do_rename() > > { > > tmpfilename=$(mktemp --dry-run) > > postfix=$(echo $tmpfilename | awk -F'/' '{print $3}') > > mv "$1" "$1-$postfix" > > } > > > > function do_dir() > > { > > for ent in $1/*; do > > if [ -d $ent ]; then > > do_dir $ent > > else > > do_rename $ent > > fi > > done > > } > > > > do_dir $mnt/testdir > > > > cd $e2fsprogs_base > > sudo umount $mnt > > sudo $fsck -f $dev > > > > exit 0 > > > > > > dumpe2fs 1.43-WIP (4-Feb-2014) > > Filesystem volume name: > > Last mounted on: /mnt/sda1 > > Filesystem UUID: 5643cf62-dd3b-4883-9991-4d7f319e3e1e > > Filesystem magic number: 0xEF53 > > Filesystem revision #: 1 (dynamic) > > Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg inline_data sparse_super large_file huge_file dir_nlink extra_isize metadata_csum > > Filesystem flags: signed_directory_hash > > Default mount options: user_xattr acl > > Filesystem state: clean > > Errors behavior: Continue > > Filesystem OS type: Linux > > Inode count: 5021696 > > Block count: 20081242 > > Reserved block count: 1004062 > > Free blocks: 19663835 > > Free inodes: 5007473 > > First block: 0 > > Block size: 4096 > > Fragment size: 4096 > > Group descriptor size: 64 > > Reserved GDT blocks: 1024 > > Blocks per group: 32768 > > Fragments per group: 32768 > > Inodes per group: 8192 > > Inode blocks per group: 512 > > Flex block group size: 16 > > Filesystem created: Wed Apr 9 11:41:36 2014 > > Last mount time: Wed Apr 9 11:41:42 2014 > > Last write time: Wed Apr 9 11:43:28 2014 > > Mount count: 0 > > Maximum mount count: -1 > > Last checked: Wed Apr 9 11:43:28 2014 > > Check interval: 0 () > > Lifetime writes: 1601 MB > > Reserved blocks uid: 0 (user root) > > Reserved blocks gid: 0 (group root) > > First inode: 11 > > Inode size: 256 > > Required extra isize: 28 > > Desired extra isize: 28 > > Journal inode: 8 > > Default directory hash: half_md4 > > Directory Hash Seed: 698d5a93-bfa8-4b55-b4ab-ae94c4e9e5e0 > > Journal backup: inode blocks > > Checksum type: crc32c > > Checksum: 0x7c4d3477 > > Journal features: journal_64bit > > Journal size: 128M > > Journal length: 32768 > > Journal sequence: 0x00000017 > > Journal start: 0 > -- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html