LinuxLists.cc - Error testing ext3 on brd ramdisk

2008-11-14 13:17:35

Subject: Error testing ext3 on brd ramdisk

Hi

I am trying to test EXT3 on a ramdisk and after some number of mounts
(anywhere from 7 to 603 so far), I get errors from e2fsck like this:

/dev/ram0: HTREE directory inode 137 has an invalid root node.
HTREE INDEX CLEARED.
/dev/ram0: Entry 'f1770' in /work/p0/da/d1c61 (137) has deleted/unused inode 692. CLEARED.
/dev/ram0: Entry 'fa37' in /work/p0/da/d1c61 (137) has deleted/unused inode 799. CLEARED.
/dev/ram0: Entry 'c1686' in /work/p0/da/d1c61 (137) has deleted/unused inode 470. CLEARED.
/dev/ram0: Entry 'c19b8' in /work/p0/da/d1c61 (137) has an incorrect filetype (was 3, should be 1).

/dev/ram0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)

This is what I do:

modprobe brd rd_nr=1 rd_size=4096
mkfs.ext3 /dev/ram0
dmesg -n8
i=0
while (true); do
i=`expr $i + 1`
echo -------------------------------------------------------------
echo Cycle $i
date
echo Mounting
sleep 1
mount -t ext3 /dev/ram0 /mnt/test_file_system || exit 1
echo Removing old fsstress data
rm -rf /mnt/test_file_system/work
mkdir /mnt/test_file_system/work || exit 1
echo Starting fsstress
fsstress -d /mnt/test_file_system/work -p 3 -l 0 -n 100000000 &
echo Sleeping 30 seconds
sleep 30
echo Stopping fsstress
while (ps -e | grep fsstress);do
pkill fsstress
sleep 1
done
echo Unmounting
umount /mnt/test_file_system || exit 1
echo Checking
sleep 1
e2fsck -fvp /dev/ram0 || exit 1
done

It seems to happen ever since the introduction of brd (Feb '08).

Does anyone know what is going on?

Regards
Adrian

2009-03-02 17:43:27

On Fri, Mar 13, 2009 at 06:06:07PM +0100, ext Jorge Boncompte [DTI2] wrote:
> Nick Piggin escribi?:
> > On Tue, Mar 10, 2009 at 05:49:13PM +0100, Jorge Boncompte [DTI2] wrote:
> >> Nick Piggin escribi?:
> >>> Ah, of course, it would be due to directory-in-pagecache. You need
> >>> the following patch if you expect this to work.
> >>>
> >>> And that confirms the ext3 problem is a different one because it
> >>> doesn't use directory in pagecache I think. Well, I'll look at
> >>> that one tomorrow.
> Hi Nick, no problems noticed in the last days running with the below
> patch...
>
> --- linux-2.6.29-rc8/fs/super.c 2009-03-13 13:46:19 +0100
> +++ kernel/fs/super.c 2009-03-13 13:47:50 +0100
> @@ -647,6 +647,8 @@ int do_remount_sb(struct super_block *sb
> acct_auto_close(sb);
> shrink_dcache_sb(sb);
> fsync_super(sb);
> + if (sb->s_bdev && (flags & MS_RDONLY))
> + invalidate_bdev(sb->s_bdev);
>
> /* If we are remounting RDONLY and current sb is read/write,
> make sure there are no rw files opened */
>

Hello,

first off, sorry if you getting this email twice.

I also tried to do ext3/ext4 fs smoketesting and used Adraian's
script. I am consistently getting the same results - filesystem get's
corrupted.
I tested on quad Xeon, with patches posted in this thread.

1. tests with brd:
- ext3fs on brd
corruption (see attached ext3fs.brd.corruption.txt)
- ext4fs on brd
corruption (see attached ext4fs.brd.corruption.txt)

In both cases I saw some complains from JBD/JBD2:
JBD: Detected IO errors while flushing file data on

2. I enabled JBD debugging, re-run the tests. Console was
flooded with messages and in the end I got a soft lockup.
I cannot consistently reproduce this (see attached
brd.ext3fs.softlock.txt).

Just to be sure I re-run the tests on real block device (usb stick)

3. tests with real block device (usb stick)
- ext3fs
no fs currption (overnight run)
- ext4fs
no fs currption (overnight run)

Any ideas what else can be done here? I'd like to find out if this is
filesystem or brd related fault.

Denis

Attachments:

(No filename) (2.10 kB)
brd.ext3fs.corruption.txt (973.00 B)
brd.ext3fs.softlock.txt (25.85 kB)
brd.ext4fs.corruption.txt (872.00 B)
Download all attachments

2009-03-18 12:15:52

by Nick Piggin

[permalink] [raw]

Subject: Re: Error testing ext3 on brd ramdisk

On Tue, Mar 17, 2009 at 11:40:19AM +0200, Denis Karpov wrote:
> Hello,
>
> first off, sorry if you getting this email twice.

No problem, I'm not exactly able to reproduce it myself, but Jan Kara
has just fixed some issues which could explain it: they happen under
memory pressure so I may not have triggered it if I didn't put it
under pressure.

Jan's fixes are here:

http://marc.info/?l=linux-ext4&m=123731584711382&w=2

It would be interesting to try them, and if they don't work maybe
he's also interested so I cc'ed him.

> I also tried to do ext3/ext4 fs smoketesting and used Adraian's
> script. I am consistently getting the same results - filesystem get's
> corrupted.
> I tested on quad Xeon, with patches posted in this thread.
>
> 1. tests with brd:
> - ext3fs on brd
> corruption (see attached ext3fs.brd.corruption.txt)
> - ext4fs on brd
> corruption (see attached ext4fs.brd.corruption.txt)
>
> In both cases I saw some complains from JBD/JBD2:
> JBD: Detected IO errors while flushing file data on
>
> 2. I enabled JBD debugging, re-run the tests. Console was
> flooded with messages and in the end I got a soft lockup.
> I cannot consistently reproduce this (see attached
> brd.ext3fs.softlock.txt).
>
> Just to be sure I re-run the tests on real block device (usb stick)
>
> 3. tests with real block device (usb stick)
> - ext3fs
> no fs currption (overnight run)
> - ext4fs
> no fs currption (overnight run)

It's possible the real block device is not fast enough to trigger
it, or different timings don't trigger it (brd requests complete
immediately wheras real devices tend to complete afterwards,
from (soft)interrupt context).

Or it could be that brd is consuming some more memory to push
the system into reclaim and exposing those bugs Jan has fixed...

> Any ideas what else can be done here? I'd like to find out if this is
> filesystem or brd related fault.

Yes, thanks for persisting. If you can test the patches and see
if they help? If not, does ext2 show corruption? How about ext3
on loop device (with backing file from tmpfs/ramfs for speed).

Thanks,
Nick

2009-03-18 13:42:21

by Jan Kara

[permalink] [raw]

Subject: Re: Error testing ext3 on brd ramdisk

> On Tue, Mar 17, 2009 at 11:40:19AM +0200, Denis Karpov wrote:
> > Hello,
> >
> > first off, sorry if you getting this email twice.
>
> No problem, I'm not exactly able to reproduce it myself, but Jan Kara
> has just fixed some issues which could explain it: they happen under
> memory pressure so I may not have triggered it if I didn't put it
> under pressure.
>
> Jan's fixes are here:
>
> http://marc.info/?l=linux-ext4&m=123731584711382&w=2
>
> It would be interesting to try them, and if they don't work maybe
> he's also interested so I cc'ed him.
>
>
> > I also tried to do ext3/ext4 fs smoketesting and used Adraian's
> > script. I am consistently getting the same results - filesystem get's
> > corrupted.
> > I tested on quad Xeon, with patches posted in this thread.
> >
> > 1. tests with brd:
> > - ext3fs on brd
> > corruption (see attached ext3fs.brd.corruption.txt)
> > - ext4fs on brd
> > corruption (see attached ext4fs.brd.corruption.txt)
> >
> > In both cases I saw some complains from JBD/JBD2:
> > JBD: Detected IO errors while flushing file data on
Yes, my patches fix exactly this problem. So please try running with
them. I'm not sure about that HTREE corruption you see during fsck. That
seems to be a separate issue.

> > 2. I enabled JBD debugging, re-run the tests. Console was
> > flooded with messages and in the end I got a soft lockup.
> > I cannot consistently reproduce this (see attached
> > brd.ext3fs.softlock.txt).
Yes, this usually produces far too many messages. The soft lockup was
probably caused by the machine being too busy logging all the messages
(log files are synced which adds much more to the load of the
filesystem). I'd probably leave that aside for now and concentrate on
the corruption problem.

Honza
--
Jan Kara <[email protected]>
SuSE CR Labs

2009-03-20 12:24:40

On Wed, Mar 18, 2009 at 02:42:02PM +0100, ext Jan Kara wrote:
> > On Tue, Mar 17, 2009 at 11:40:19AM +0200, Denis Karpov wrote:
> > Jan's fixes are here:
> > http://marc.info/?l=linux-ext4&m=123731584711382&w=2
> > It would be interesting to try them, and if they don't work maybe
> > he's also interested so I cc'ed him.

Hello,

I've re-run the tests (with Jan's patches and also Nick's "fs: new inode
i_state corruption fix" patch).

> > > In both cases I saw some complains from JBD/JBD2:
> > > JBD: Detected IO errors while flushing file data on
> Yes, my patches fix exactly this problem. So please try running with
> them. I'm not sure about that HTREE corruption you see during fsck. That
> seems to be a separate issue.

The issue with JBD seems to be gone. But problem with HTREE being corrupted
still remains (see attached logs).

Denis

Attachments:

(No filename) (852.00 B)
ext3.htree.1.txt (5.12 kB)
ext4.htree.1.txt (2.13 kB)
Download all attachments