On Tue, Sep 09, 2008 at 05:55:31PM -0400, Theodore Tso wrote:
> > > Unfortunately this is one of those bugs that I can't find a way to
> > > reproduce except by randomly breaking one fs after another. This
> > > happens with ext3 and ext4, but so far I haven't seen it happen
> > > with ext2.
> > >
> > >
> > > *** seed 270, ext3, 2.6.27-rc3 ***
> > > *** seed 451, ext4, 2.6.27-rc5 ***
>
> Given these seed numbers, I assume this was generating using some tool
> like fsfuzzer? Would it be possible to generate a filesystem image
> *before* that triggers the problem case, before trying to execute the
> rm -rf?
>
> That would be the fastest way to try to track the problem down.
Yes, I can generate those filesystems. However the problem seems to be
elusive in that I haven't yet been able to reproduce it twice with the
same filesystem (and even with random filesystems, it every occurs
once in a while). I'll do some more testing and try to figure out if
it can be reproduced more easily. Still I can give you some
filesystems that crashed once, if you wish. They are typically
something like 600 KiB compressed, and I guess that could be made less
by zeroing all regular files in the pristine fs before doing the
fuzzing.
Here's a script I use to do the testing ($1 is the initial seed). The
filesystem is a 10 MiB pristine ext[34] image with a copy of my
workstation's /dev and a partial copy of /usr/share/doc (I tried to be
diverse in what I put there).
------------------------------------------------------------
#!/bin/sh
if [ "`hostname`" != "fstest" ]; then
echo "This is a dangerous script."
echo "Set your hostname to \`fstest\' if you want to use it."
exit 1
fi
umount /dev/hdb
umount /dev/hdc
/etc/init.d/sysklogd stop
/etc/init.d/klogd stop
/etc/init.d/cron stop
mount /dev/hda / -t ext3 -o remount,ro || exit 1
#ulimit -t 20
for ((s=$1; s<1000000000; s++)); do
umount /mnt
echo '***** zzuffing *****' seed $s
zzuf -r 0:0.03 -s $s </dev/hdc >/dev/hdb || exit
mount /dev/hdb /mnt -t ext2 -o errors=continue || continue
cd /mnt || continue
timeout 30 cp -r doc doc2 >&/dev/null
timeout 30 find -xdev >&/dev/null
timeout 30 find -xdev -print0 2>/dev/null |xargs -0 touch -- 2>/dev/null
timeout 30 mkdir tmp >&/dev/null
timeout 30 echo whoah >tmp/filu 2>/dev/null
timeout 30 rm -rf /mnt/* >&/dev/null
cd /
done
------------------------------------------------------------
Sami
On Wed, Sep 10, 2008 at 06:26:34AM +0300, Sami Liedes wrote:
>
> Yes, I can generate those filesystems. However the problem seems to be
> elusive in that I haven't yet been able to reproduce it twice with the
> same filesystem (and even with random filesystems, it every occurs
> once in a while). I'll do some more testing and try to figure out if
> it can be reproduced more easily. Still I can give you some
> filesystems that crashed once, if you wish. They are typically
> something like 600 KiB compressed, and I guess that could be made less
> by zeroing all regular files in the pristine fs before doing the
> fuzzing.
One easy way of doing this is the following:
e2image -r /dev/hdXX /var/tmp/hdXX.e2i
dd if=/var/tmp/hdXX.e2i of=/dev/hdXX
Another thing you can do is change your script to add the following
line before the filesystem is mounted:
e2image -r /dev/hdXX - | bzip2 > /var/tmp/hdXX.e2i
and then if the filesystem fails (i.e., the system oops),
/var/tmp/hdXX.e2i.bz2 will have all of the filesystem metadata
(including directories), such that if you decompress and write out the
filesystem (or what I do when given one of these to examine):
bunzip2 < hdXX.e2i.bz2 | make-sparse > hdXX.e2i
Said sparse file can now be checked via e2fsck, or mounted using a
loopback mount, etc.
Even if it's not reliably reproducable, if I can get a series of
filesystems which show the problem, using "e2fsck -nf" we can see a
pattern of how the filesystems are corrupted, and that can help narrow
down what might be going on that causes the kernel oops.
Thanks, regards,
- Ted
/*
* make-sparse.c --- make a sparse file from stdin
*
* Copyright 2004 by Theodore Ts'o.
*
* %Begin-Header%
* This file may be redistributed under the terms of the GNU Public
* License.
* %End-Header%
*/
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
int full_read(int fd, char *buf, size_t count)
{
int got, total = 0;
int pass = 0;
while (count > 0) {
got = read(fd, buf, count);
if (got == -1) {
if ((errno == EINTR) || (errno == EAGAIN))
continue;
return total ? total : -1;
}
if (got == 0) {
if (pass++ >= 3)
return total;
continue;
}
pass = 0;
buf += got;
total += got;
count -= got;
}
return total;
}
int main(int argc, char **argv)
{
int fd, got, i;
char buf[1024];
if (argc != 2) {
fprintf(stderr, "Usage: make-sparse out-file\n");
exit(1);
}
fd = open(argv[1], O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0777);
if (fd < 0) {
perror(argv[1]);
exit(1);
}
while (1) {
got = full_read(0, buf, sizeof(buf));
if (got == 0)
break;
if (got == sizeof(buf)) {
for (i=0; i < sizeof(buf); i++)
if (buf[i])
break;
if (i == sizeof(buf)) {
lseek(fd, sizeof(buf), SEEK_CUR);
continue;
}
}
write(fd, buf, got);
}
return 0;
}