LinuxLists.cc - data corruption with ext4 (from 2.6.27.4) exposed by rtorrent

2008-11-03 12:42:12

Subject: data corruption with ext4 (from 2.6.27.4) exposed by rtorrent

Some person already reported this. I'm having the same problem.
http://marc.info/?l=linux-ext4&m=122557056518246&w=2
rtorrent is a bittorrent client that makes heavy use of mmap instead
of read/write to avoid needless duplication of data. It has exposed
other bugs in the past (mmap bug in 2.6.19).
http://libtorrent.rakshasa.no
Downloading a big torrent (>2GB) with rtorrent triggers it more times
than not. The bigger the torrent, the higher the likeliness of
failure. When the torrent is finished, if a hash check is forced by
pressing control-r on the torrent, some blocks will fail.
The torrent doesn't need to be a single file, but can be a series of
small files that add up enough.
Downloading on ext3 or XFS instead is 100% fine.
I've discarded posix fallocate as the culprit by building rtorrent
without the patch that adds that (and doing a du just in case,
definitively not preallocating).
Filesystem is ~1.4TB. I've also reproduced it on a 200GB partition.
The system is: Linux kaguya 2.6.27.4 #1 SMP Mon Oct 27 14:43:19 UTC
2008 x86_64 Intel(R) Atom(TM) CPU 330 @ 1.60GHz GenuineIntel GNU/Linux
No kernel patches were applied.

2008-11-03 13:40:10

by Theodore Ts'o

[permalink] [raw]

Subject: Re: data corruption with ext4 (from 2.6.27.4) exposed by rtorrent

On Mon, Nov 03, 2008 at 12:42:11PM +0000, Roc Valles wrote:
> Some person already reported this. I'm having the same problem.
> http://marc.info/?l=linux-ext4&m=122557056518246&w=2
>
> rtorrent is a bittorrent client that makes heavy use of mmap instead
> of read/write to avoid needless duplication of data. It has exposed
> other bugs in the past (mmap bug in 2.6.19).
> http://libtorrent.rakshasa.no
> Downloading a big torrent (>2GB) with rtorrent triggers it more times
> than not. The bigger the torrent, the higher the likeliness of
> failure. When the torrent is finished, if a hash check is forced by
> pressing control-r on the torrent, some blocks will fail.

Can both of you send the output of "dumpe2fs -h /dev/<disk device>" of
the filesystem in question? The thing which I'm most interested in is
whether the extents feature was enabled or not. (i.e., was this a
freshly made ext4 filesystem, or a ext3 filesystem mounted under ext4,
and with which features eanbled?)

Thanks,

- Ted

2008-11-03 13:47:12

by Cryptooctoploid

[permalink] [raw]

Subject: Re: data corruption with ext4 (from 2.6.27.4) exposed by rtorrent

On Mon, Nov 3, 2008 at 2:40 PM, Theodore Tso <[email protected]> wrote:
> On Mon, Nov 03, 2008 at 12:42:11PM +0000, Roc Valles wrote:
>> Some person already reported this. I'm having the same problem.
>> http://marc.info/?l=linux-ext4&m=122557056518246&w=2
>>
>> rtorrent is a bittorrent client that makes heavy use of mmap instead
>> of read/write to avoid needless duplication of data. It has exposed
>> other bugs in the past (mmap bug in 2.6.19).
>> http://libtorrent.rakshasa.no
>> Downloading a big torrent (>2GB) with rtorrent triggers it more times
>> than not. The bigger the torrent, the higher the likeliness of
>> failure. When the torrent is finished, if a hash check is forced by
>> pressing control-r on the torrent, some blocks will fail.
>
> Can both of you send the output of "dumpe2fs -h /dev/<disk device>" of
> the filesystem in question? The thing which I'm most interested in is
> whether the extents feature was enabled or not. (i.e., was this a
> freshly made ext4 filesystem, or a ext3 filesystem mounted under ext4,
> and with which features eanbled?)
>

OK. In my case it's a freshly made default ext4 filesystem.
I'm running 2.6.28-rc3 x86_64.

dumpe2fs 1.41.3 (12-Oct-2008)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 511e7636-5196-4095-8654-bca756102d8d
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent flex_bg sparse_super large_file
huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 6406144
Block count: 25599569
Reserved block count: 1279978
Free blocks: 19257546
Free inodes: 5623009
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1017
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Fri Oct 24 15:48:46 2008
Last mount time: Mon Nov 3 09:59:24 2008
Last write time: Mon Nov 3 09:59:24 2008
Mount count: 20
Maximum mount count: 28
Last checked: Sat Nov 1 20:07:13 2008
Check interval: 15552000 (6 months)
Next check after: Thu Apr 30 21:07:13 2009
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 294551
Default directory hash: half_md4
Directory Hash Seed: 847df4ea-f163-404b-bc41-07a5a5245911
Journal backup: inode blocks
Journal size: 128M

2008-11-03 14:09:36

by Roc Valles

[permalink] [raw]

Subject: Re: data corruption with ext4 (from 2.6.27.4) exposed by rtorrent

Ok. It's a recently made ext4 using mkfs.ext4dev from latest
sysresccd. Inmediatly after making it, I moved in my root from an old
500GB ext3 hard disk.

This is the output:

kaguya ~ # dumpe2fs -h /dev/sda3
dumpe2fs 1.41.3 (12-Oct-2008)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 07520665-bfed-4d37-a38e-0a7cb04cff59
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent flex_bg sparse_super large_file
huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash test_filesystem
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 91430912
Block count: 365693619
Reserved block count: 0
Free blocks: 245016006
Free inodes: 90916075
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 936
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Fri Oct 31 13:43:31 2008
Last mount time: Mon Nov 3 11:17:20 2008
Last write time: Mon Nov 3 11:17:20 2008
Mount count: 5
Maximum mount count: 31
Last checked: Fri Oct 31 13:43:31 2008
Check interval: 15552000 (6 months)
Next check after: Wed Apr 29 13:43:31 2009
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: ba0cc33f-c339-4fd9-aa04-bfdad45ecf92
Journal backup: inode blocks
Journal size: 128M

On Mon, Nov 3, 2008 at 13:40, Theodore Tso <[email protected]> wrote:
> Can both of you send the output of "dumpe2fs -h /dev/<disk device>" of
> the filesystem in question? The thing which I'm most interested in is
> whether the extents feature was enabled or not. (i.e., was this a
> freshly made ext4 filesystem, or a ext3 filesystem mounted under ext4,
> and with which features eanbled?)
>
> Thanks,
>
> - Ted

2008-11-03 15:34:12

by Cryptooctoploid

[permalink] [raw]

[email protected] writes:
> Aneesh Kumar K.V writes:
> > The patch aneesh-9-fix-lockdeep-recursive-locking-warning
> > should add the key in .data. Do you have a stack trace ?
>
> [ 31.068245] kjournald2 starting. Commit interval 5 seconds
> [ 31.139463] EXT4 FS on hdb, internal journal on hdb:8
> [ 31.201526] EXT4-fs: delayed allocation enabled
> [ 31.253465] EXT4-fs: file extents enabled
> [ 31.305491] BUG: key ca85b5a8 not in .data!
> [ 31.357287] ------------[ cut here ]------------
> [ 31.409201] WARNING: at kernel/lockdep.c:2499 __init_rwsem+0x21/0x47()
> [ 31.481599] Modules linked in: ext4 jbd2 crc16 video output ac battery dm_snapshot dm_mirror dm_region_hash dm_log dm_mod loop serio_raw pcspkr psmouse i2c_piix4 i2c_core parport_pc parport button evdev ext3 jbd mbcache ide_cd_mod cdrom ide_gd_mod ata_generic ata_piix libata scsi_mod ide_pci_generic floppy 8139too piix 8139cp mii ide_core thermal processor fan thermal_sys
> [ 31.873748] Pid: 2032, comm: mount Not tainted 2.6.28-rc3-ext4-patch-queue #1
> [ 31.966325] Call Trace:
> [ 31.987733] [<c01299dd>] warn_on_slowpath+0x40/0x61
> [ 32.049842] [<c0129fcd>] release_console_sem+0x19f/0x1cc
> [ 32.111988] [<c012a48c>] vprintk+0x2dd/0x309
> [ 32.163924] [<c0188209>] cache_alloc_refill+0x19f/0x450
> [ 32.226049] [<c02d9179>] printk+0xe/0x15
> [ 32.277932] [<c0145353>] lockdep_init_map+0x8d/0x420
> [ 32.340000] [<c0146bea>] debug_check_no_locks_freed+0x110/0x123
> [ 32.412332] [<c0146aab>] trace_hardirqs_on_caller+0xed/0x114
> [ 32.474575] [<c020194e>] __init_rwsem+0x21/0x47
> [ 32.536589] [<ca84807b>] ext4_mb_add_groupinfo+0x117/0x146 [ext4]
> [ 32.609828] [<ca8482b4>] ext4_mb_init+0x20a/0x60e [ext4]
> [ 32.671977] [<ca83ab2a>] ext4_fill_super+0x19d6/0x1f47 [ext4]
> [ 32.744330] [<c0118c76>] pvclock_clocksource_read+0x48/0xa7
> [ 32.806561] [<c0202cc3>] snprintf+0x16/0x18
> [ 32.858464] [<c01c25f7>] disk_name+0x1f/0x5b
> [ 32.910343] [<c018d44f>] get_sb_bdev+0xeb/0x12c
> [ 32.962300] [<c0146aab>] trace_hardirqs_on_caller+0xed/0x114
> [ 33.034576] [<ca8376d9>] ext4_get_sb+0x12/0x16 [ext4]
> [ 33.096667] [<ca839154>] ext4_fill_super+0x0/0x1f47 [ext4]
> [ 33.158809] [<c018d11c>] vfs_kern_mount+0x81/0xf3
> [ 33.220857] [<c018d1cc>] do_kern_mount+0x2f/0xb4
> [ 33.272860] [<c019e2cf>] do_mount+0x63f/0x67a
> [ 33.324749] [<c018802b>] kmem_cache_alloc+0x85/0xc4
> [ 33.386752] [<c019e379>] sys_mount+0x6f/0xaa
> [ 33.438659] [<c0103ac6>] syscall_call+0x7/0xb
> [ 33.490525] ---[ end trace 4cec533a62581305 ]---
> [ 33.546858] EXT4-fs: mballoc enabled
> [ 33.588549] EXT4-fs: mounted filesystem with ordered data mode.

I am not observing it outside of kvm.

--
solofo

2008-11-06 14:00:08

by Aneesh Kumar K.V

[permalink] [raw]

Subject: Re: data corruption with ext4 (from 2.6.27.4) exposed by rtorrent

On Wed, Nov 05, 2008 at 08:44:32PM +0100, [email protected] wrote:
> Hi Jindrich,
>
> Jindrich Makovicka writes:
> > The following testcase was used to trigger the infamous MAP_SHARED
> > dirty flag bug. Maybe it could be of some help here too:
> >
> > http://lkml.org/lkml/2006/12/27/180
>
> Thanks for remembering!
>
> The test case triggers corruption with 2.6.28-rc3 + ext4 patch queue:
> . bunch of errors like
> Chunk 71637 corrupted (0-1339) (2756-4095)
> Expected 213, got 0
> with default mount.
> . nodelalloc is ok.

I think below patch is needed. But even with the patch I am hitting the
corruption. So patch doesn't solve the problem. The change from tag
dirty lookup to index lookup should not have much impact because we are
writing from first_page to next_page - 1 contiguous page which should be
marked dirty anyway because we did a tag dirty lookup in
write_cache_pages and we did a lock page there.

commit 714736001d68cff26258bb80891c566a65a682c2
Author: Aneesh Kumar K.V <[email protected]>
Date: Thu Nov 6 09:01:21 2008 +0530

ext4: Fix the delalloc writepages to allocate blocks at the right offset.

When iterating through the pages with all mapped buffer_heads
we failed to update the b_state value. This result in allocating
blocks at logical offset 0.

Signed-off-by: Aneesh Kumar K.V <[email protected]>

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 7d2d2ea..2808984 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1667,35 +1667,41 @@ static void ext4_da_page_release_reservation(struct page *page,
*/
static int mpage_da_submit_io(struct mpage_da_data *mpd)
{
- struct address_space *mapping = mpd->inode->i_mapping;
- int ret = 0, err, nr_pages, i;
- unsigned long index, end;
- struct pagevec pvec;
long pages_skipped;
+ struct pagevec pvec;
+ unsigned long index, end;
+ int ret = 0, err, nr_pages, i;
+ struct inode *inode = mpd->inode;
+ struct buffer_head *lbh = &mpd->lbh;
+ struct address_space *mapping = inode->i_mapping;

BUG_ON(mpd->next_page <= mpd->first_page);
- pagevec_init(&pvec, 0);
+ /*
+ * we need to start from the first_page to the next_page - 1
+ * That is to make sure we also write the mapped dirty
+ * buffer_heads. If we look at mpd->lbh.b_blocknr we
+ * would only be looking at currently mapped buffer_heads.
+ */
index = mpd->first_page;
end = mpd->next_page - 1;

+ pagevec_init(&pvec, 0);
while (index <= end) {
- /*
- * We can use PAGECACHE_TAG_DIRTY lookup here because
- * even though we have cleared the dirty flag on the page
- * We still keep the page in the radix tree with tag
- * PAGECACHE_TAG_DIRTY. See clear_page_dirty_for_io.
- * The PAGECACHE_TAG_DIRTY is cleared in set_page_writeback
- * which is called via the below writepage callback.
- */
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
- PAGECACHE_TAG_DIRTY,
- min(end - index,
- (pgoff_t)PAGEVEC_SIZE-1) + 1);
+ nr_pages = pagevec_lookup(&pvec, mapping, index, PAGEVEC_SIZE);
if (nr_pages == 0)
break;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];

+ index = page->index;
+ if (index > end)
+ break;
+ index++;
+
+ BUG_ON(!PageLocked(page));
+ BUG_ON(PageWriteback(page));
+ BUG_ON(!page_has_buffers(page));
+
pages_skipped = mpd->wbc->pages_skipped;
err = mapping->a_ops->writepage(page, mpd->wbc);
if (!err && (pages_skipped == mpd->wbc->pages_skipped))
@@ -2109,11 +2115,29 @@ static int __mpage_da_writepage(struct page *page,
bh = head;
do {
BUG_ON(buffer_locked(bh));
+ /*
+ * We need to try to allocte
+ * unmapped blocks in the same page.
+ * Otherwise we won't make progress
+ * with the page in ext4_da_writepage
+ */
if (buffer_dirty(bh) &&
(!buffer_mapped(bh) || buffer_delay(bh))) {
mpage_add_bh_to_extent(mpd, logical, bh);
if (mpd->io_done)
return MPAGE_DA_EXTENT_TAIL;
+ } else if (buffer_dirty(bh) && (buffer_mapped(bh))) {
+ /*
+ * mapped dirty buffer. We need to update
+ * the b_state because we look at
+ * b_state in mpage_da_map_blocks. We don't
+ * update b_size because if we find an
+ * unmapped buffer_head later we need to
+ * use the b_state flag of that buffer_head.
+ */
+ if (mpd->lbh.b_size == 0)
+ mpd->lbh.b_state =
+ bh->b_state & BH_FLAGS;
}
logical++;
} while ((bh = bh->b_this_page) != head);