2003-08-29 17:24:52

by Mike Fedyk

[permalink] [raw]
Subject: mutt segfault with ext3 & 1k blocks & htree in 2.6

Hi,

I have just converted my 25GB / partition from reiserfs to ext3 with 1k
blocks, and now mutt is segfaulting periodocally.

I suspect it is htree, because I left four mutt processes running last
night, and two of them segfaulted.

Interestingly enough, it happened about 1 minute apart, so they might have
checked one of the large maildir folders and that could have caused the
problem, except that the other two mutt processes should have checked the
same folders, and they didn't crash.

I have full strace output of each mutt process up until the segfault in two
cases, and up until strace was stopped in the third case.

Please let me know what more I can do to help track this down.

I have tried this with:
vmlinuz-2.6.0-test3-mm3
vmlinuz-2.6.0-test4-mm1


2003-08-29 18:09:58

by Mike Fedyk

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Fri, Aug 29, 2003 at 10:24:51AM -0700, Mike Fedyk wrote:
> Hi,
>
> I have just converted my 25GB / partition from reiserfs to ext3 with 1k
> blocks, and now mutt is segfaulting periodocally.
>
> I suspect it is htree, because I left four mutt processes running last
> night, and two of them segfaulted.
>
> Interestingly enough, it happened about 1 minute apart, so they might have
> checked one of the large maildir folders and that could have caused the
> problem, except that the other two mutt processes should have checked the
> same folders, and they didn't crash.
>
> I have full strace output of each mutt process up until the segfault in two
> cases, and up until strace was stopped in the third case.
>
> Please let me know what more I can do to help track this down.
>
> I have tried this with:
> vmlinuz-2.6.0-test3-mm3
> vmlinuz-2.6.0-test4-mm1
>

How do I:

o Find out that a directory is using htree?
o Disable htree on my /? (tune2fs -O ^dir_index), but then how do I get my
directories back to non-htree without running fsck from a rescue CD?

Thanks,

Mike

2003-08-29 18:33:25

by Andy Isaacson

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Fri, Aug 29, 2003 at 10:24:51AM -0700, Mike Fedyk wrote:
> I have just converted my 25GB / partition from reiserfs to ext3 with 1k
> blocks, and now mutt is segfaulting periodocally.
[snip]
> I have full strace output of each mutt process up until the segfault in two
> cases, and up until strace was stopped in the third case.

The obvious request is "turn on core dumps and get a backtrace".

-andy

2003-08-29 19:00:53

by Mike Fedyk

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Fri, Aug 29, 2003 at 01:33:23PM -0500, Andy Isaacson wrote:
> On Fri, Aug 29, 2003 at 10:24:51AM -0700, Mike Fedyk wrote:
> > I have just converted my 25GB / partition from reiserfs to ext3 with 1k
> > blocks, and now mutt is segfaulting periodocally.
> [snip]
> > I have full strace output of each mutt process up until the segfault in two
> > cases, and up until strace was stopped in the third case.
>
> The obvious request is "turn on core dumps and get a backtrace".

The problem only shows on 2.6, and it works perfectly on 2.4. I don't think
it's an app issue. Also, Debian doesn't compile in the debug symbols, so
the backtrace is not very much use. (though, I can show the output if you'd
like, just give me another day for it to segfault again.)

Mike

2003-08-30 19:15:11

by Andreas Dilger

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Aug 29, 2003 11:09 -0700, Mike Fedyk wrote:
> On Fri, Aug 29, 2003 at 10:24:51AM -0700, Mike Fedyk wrote:
> o Find out that a directory is using htree?

"lsattr <dir>" will show it. Note that it will only ever be set on
directories that are larger than a single disk block.

# lsattr -d d1
----------I-- d1

> o Disable htree on my /? (tune2fs -O ^dir_index), but then how do I get
> my directories back to non-htree without running fsck from a rescue CD?

That's the great thing about htree - you don't need to do anything to turn
it off. The on-disk format is exactly the same as without htree, and the
first time you modify the directory it will clear the per-directory htree
flag.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

2003-08-30 23:58:19

by Mike Fedyk

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Sat, Aug 30, 2003 at 01:14:21PM -0600, Andreas Dilger wrote:
> On Aug 29, 2003 11:09 -0700, Mike Fedyk wrote:
> > On Fri, Aug 29, 2003 at 10:24:51AM -0700, Mike Fedyk wrote:
> > o Find out that a directory is using htree?
>
> "lsattr <dir>" will show it. Note that it will only ever be set on
> directories that are larger than a single disk block.
>
> # lsattr -d d1
> ----------I-- d1
>

Ok, now I only have htree enabled on one of my maildir folders.

> > o Disable htree on my /? (tune2fs -O ^dir_index), but then how do I get
> > my directories back to non-htree without running fsck from a rescue CD?
>
> That's the great thing about htree - you don't need to do anything to turn
> it off. The on-disk format is exactly the same as without htree, and the
> first time you modify the directory it will clear the per-directory htree
> flag.

I'll do more testing to see if it fails only on that folder now.

But how do I re-enable htree on the directories (besides an fsck -D) in a
live system?

2003-08-31 22:45:35

by Andreas Dilger

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Aug 30, 2003 16:58 -0700, Mike Fedyk wrote:
> On Sat, Aug 30, 2003 at 01:14:21PM -0600, Andreas Dilger wrote:
> > On Aug 29, 2003 11:09 -0700, Mike Fedyk wrote:
> > > On Fri, Aug 29, 2003 at 10:24:51AM -0700, Mike Fedyk wrote:
> > > o Find out that a directory is using htree?
> >
> > "lsattr <dir>" will show it. Note that it will only ever be set on
> > directories that are larger than a single disk block.
> >
> > # lsattr -d d1
> > ----------I-- d1
> >
>
> Ok, now I only have htree enabled on one of my maildir folders.
>
> > > o Disable htree on my /? (tune2fs -O ^dir_index), but then how do I get
> > > my directories back to non-htree without running fsck from a rescue CD?
> >
> > That's the great thing about htree - you don't need to do anything to turn
> > it off. The on-disk format is exactly the same as without htree, and the
> > first time you modify the directory it will clear the per-directory htree
> > flag.
>
> I'll do more testing to see if it fails only on that folder now.
>
> But how do I re-enable htree on the directories (besides an fsck -D) in a
> live system?

You need to re-enable the dir_index feature, and then for directories which
are larger than a block in size you need something like:

mkdir new_dir
mv old_dir/* new_dir
rmdir old_dir
mv new_dir old_dir

The new directory will have htree enabled because it started out at 1 block
in size.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

2003-09-01 20:27:22

by Mike Fedyk

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Sun, Aug 31, 2003 at 04:44:48PM -0600, Andreas Dilger wrote:
> On Aug 30, 2003 16:58 -0700, Mike Fedyk wrote:
> > But how do I re-enable htree on the directories (besides an fsck -D) in a
> > live system?
>
> You need to re-enable the dir_index feature, and then for directories which
> are larger than a block in size you need something like:
>
> mkdir new_dir
> mv old_dir/* new_dir
> rmdir old_dir
> mv new_dir old_dir
>
> The new directory will have htree enabled because it started out at 1 block
> in size.

Ok I ended up doing this after a little thought. Thanks.

But I am seeing segfaults in mutt under 2.6 ext3 with 1k blocks, that I
don't see in 2.4, and didn't see under reiserfs. I can try with 4k blocks,
if you'd like, is there anything I can do to capture more information that
could be happening to cause this?

And now mutt is segfaulting on non-htree directories too.

2003-09-02 16:10:30

by Andreas Dilger

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Sep 01, 2003 13:27 -0700, Mike Fedyk wrote:
> On Sun, Aug 31, 2003 at 04:44:48PM -0600, Andreas Dilger wrote:
> > On Aug 30, 2003 16:58 -0700, Mike Fedyk wrote:
> > > But how do I re-enable htree on the directories (besides an fsck -D) in a
> > > live system?
> >
> > You need to re-enable the dir_index feature, and then for directories which
> > are larger than a block in size you need something like:
> >
> > mkdir new_dir
> > mv old_dir/* new_dir
> > rmdir old_dir
> > mv new_dir old_dir
> >
> > The new directory will have htree enabled because it started out at 1 block
> > in size.
>
> Ok I ended up doing this after a little thought. Thanks.
>
> But I am seeing segfaults in mutt under 2.6 ext3 with 1k blocks, that I
> don't see in 2.4, and didn't see under reiserfs. I can try with 4k blocks,
> if you'd like, is there anything I can do to capture more information that
> could be happening to cause this?

Normally an application segfault is really caused by a kernel OOPS, so if
you look into your syslog file or dmesg output you should see an oops.

> And now mutt is segfaulting on non-htree directories too.

I couldn't comment on that, but either the directories are somehow corrupted
(e2fsck will know), or the problem is related either to 1kB blocks or not
related to the filesystem at all.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

2003-09-02 21:58:32

by Mike Fedyk

[permalink] [raw]
Subject: Re: mutt segfault with ext3 & 1k blocks & htree in 2.6

On Tue, Sep 02, 2003 at 10:09:27AM -0600, Andreas Dilger wrote:
> Normally an application segfault is really caused by a kernel OOPS, so if
> you look into your syslog file or dmesg output you should see an oops.
>

No, I'm not seeing any oopses on this machine at all right now (not that I
couldn't cause a couple at will, but those are reported with varying degrees
of success on getting fixes)

> > And now mutt is segfaulting on non-htree directories too.
>
> I couldn't comment on that, but either the directories are somehow corrupted
> (e2fsck will know), or the problem is related either to 1kB blocks or not
> related to the filesystem at all.

Ok, I will convert my ext3 (cp twice) to 4k blocks, and try to reproduce.
Hopefully I didn't hit a library update (in debian a mixed debian testing
/ unstable system) that caused this problem.