I have an ext3 filesystem that has been having problems
with its journal. The result is that the file system
remounts internally as read-only and the server becomes
unusable, even shutdown does not work, using up 100% of
the CPU but not rebooting.
I found some postings indicating that mounting it as
ext2 should fix the problem, as it doesn't appear to be
a hardware issue.
So, I decided to mount everything as ext2. Mount shows this:
# mount
/dev/hda2 on / type ext2 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext2 (rw)
none on /dev/shm type tmpfs (rw,noexec)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
But now I still get the error:
# dmesg
[...]
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
[...]
The kernel is:
# uname -a
Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004
i686 i686 i386 GNU/Linux
Any ideas?
Thanks,
Alberto
--
Alberto Alonso Global Gate Systems LLC.
(512) 351-7233 http://www.ggsys.net
Hardware, consulting, sysadmin, monitoring and remote backups
On Thu, 11 Jan 2007 23:08:16 -0600
Alberto Alonso <[email protected]> wrote:
> I have an ext3 filesystem that has been having problems
> with its journal. The result is that the file system
> remounts internally as read-only and the server becomes
> unusable, even shutdown does not work, using up 100% of
> the CPU but not rebooting.
>
> I found some postings indicating that mounting it as
> ext2 should fix the problem, as it doesn't appear to be
> a hardware issue.
>
> So, I decided to mount everything as ext2. Mount shows this:
>
> # mount
> /dev/hda2 on / type ext2 (rw,usrquota)
> none on /proc type proc (rw)
> none on /sys type sysfs (rw)
> none on /dev/pts type devpts (rw,gid=5,mode=620)
> usbfs on /proc/bus/usb type usbfs (rw)
> /dev/hda1 on /boot type ext2 (rw)
> none on /dev/shm type tmpfs (rw,noexec)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>
> But now I still get the error:
>
> # dmesg
> [...]
> EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> [...]
>
>
> The kernel is:
>
> # uname -a
> Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004
> i686 i686 i386 GNU/Linux
>
>
> Any ideas?
>
mount(8) tells lies. Look in /proc/mounts and you'll see that it's really
mounted as ext3.
You probably want to add `rootfstype=ext2' to the kernel boot command line.
You were right, even after making the changes, it seems to be
telling lies:
# mount
/dev/hda2 on / type ext2 (rw,usrquota)
[...]
However, I think I am still not mounting as ext2:
# dmesg | grep 'Kernel command'
Kernel command line: ro root=/dev/hda2 rootfstype=ext2
# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw,nodiratime 0 0
/sys /sys sysfs rw 0 0
none /dev/pts devpts rw 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/hda1 /boot ext2 rw 0 0
none /dev/shm tmpfs rw,noexec 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
Do I need to mess with the initrd? My grub lines look like
this:
title Fedora Core (2.6.5-1.358smp)
root (hd0,0)
kernel /vmlinuz-2.6.5-1.358smp ro root=/dev/hda2 rootfstype=ext2
initrd /initrd-2.6.5-1.358smp.img
title Fedora Core-up (2.6.5-1.358)
root (hd0,0)
kernel /vmlinuz-2.6.5-1.358 ro root=/dev/hda2 rootfstype=ext2
initrd /initrd-2.6.5-1.358.img
Thanks,
Alberto
On Thu, 2007-01-11 at 21:25 -0800, Andrew Morton wrote:
> On Thu, 11 Jan 2007 23:08:16 -0600
> Alberto Alonso <[email protected]> wrote:
>
> > I have an ext3 filesystem that has been having problems
> > with its journal. The result is that the file system
> > remounts internally as read-only and the server becomes
> > unusable, even shutdown does not work, using up 100% of
> > the CPU but not rebooting.
> >
> > I found some postings indicating that mounting it as
> > ext2 should fix the problem, as it doesn't appear to be
> > a hardware issue.
> >
> > So, I decided to mount everything as ext2. Mount shows this:
> >
> > # mount
> > /dev/hda2 on / type ext2 (rw,usrquota)
> > none on /proc type proc (rw)
> > none on /sys type sysfs (rw)
> > none on /dev/pts type devpts (rw,gid=5,mode=620)
> > usbfs on /proc/bus/usb type usbfs (rw)
> > /dev/hda1 on /boot type ext2 (rw)
> > none on /dev/shm type tmpfs (rw,noexec)
> > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> >
> > But now I still get the error:
> >
> > # dmesg
> > [...]
> > EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> > EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> > EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> > EXT3-fs error (device hda2) in start_transaction: Journal has aborted
> > [...]
> >
> >
> > The kernel is:
> >
> > # uname -a
> > Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004
> > i686 i686 i386 GNU/Linux
> >
> >
> > Any ideas?
> >
>
> mount(8) tells lies. Look in /proc/mounts and you'll see that it's really
> mounted as ext3.
>
> You probably want to add `rootfstype=ext2' to the kernel boot command line.
>
--
Alberto Alonso Global Gate Systems LLC.
(512) 351-7233 http://www.ggsys.net
Hardware, consulting, sysadmin, monitoring and remote backups
Hi!
> You were right, even after making the changes, it seems to be
> telling lies:
>
> # mount
> /dev/hda2 on / type ext2 (rw,usrquota)
> [...]
>
> However, I think I am still not mounting as ext2:
>
> # dmesg | grep 'Kernel command'
> Kernel command line: ro root=/dev/hda2 rootfstype=ext2
...
> rootfs / rootfs rw 0 0
> /dev/root / ext3 rw 0 0
> Do I need to mess with the initrd? My grub lines look like
> this:
Yes, probably.
Pavel
--
Thanks for all the (sleeping) penguins.
Why not just use tune2fs to remove the ext3 journal?
>> You were right, even after making the changes, it seems to be
>> telling lies:
>>
>> # mount
>> /dev/hda2 on / type ext2 (rw,usrquota)
Roughly speaking:
/etc/mtab shows you what you said to mount.
/proc/mounts shows what the current kernel state is.
These may differ greatly.
For all filesystems mounted by you using mount(8), a line is added
to /etc/mtab, where the contents of that line is related to the
given mount command, but not to what the kernel did.
For the root filesystem, mount(8) writes an initial line in /etc/mtab
taken from /etc/fstab. Again the information is from you, not from the kernel.
>> # dmesg | grep 'Kernel command'
>> Kernel command line: ro root=/dev/hda2 rootfstype=ext2
> ...
>> /dev/root / ext3 rw 0 0
It would be a bad bug if the kernel mounted its root filesystem
with a type different from the type given in "rootfstype=".
But I see you use an initrd, and there can be all kinds of commands there.
Thank you all for all your input. The tune2fs option was eventually
used and we run into other problems. I think Andries was right in
that the initrd was interfering, that's where we run into issues
after the tune2fs.
I was trying to avoid the tune2fs as it involves booting into
a live CD and brings the system down to where I can't access
it over the network (it is a 4 hour drive).
At the end we had to replace the drive and recreate all file
systems. If it ever happens again I will pay closer attention
to the initrd commands to see if the rootfstype=ext2 was overridden
with what's there.
Thanks,
Alberto
On Thu, 2007-01-18 at 23:05 +0100, Andries Brouwer wrote:
> >> You were right, even after making the changes, it seems to be
> >> telling lies:
> >>
> >> # mount
> >> /dev/hda2 on / type ext2 (rw,usrquota)
>
> Roughly speaking:
> /etc/mtab shows you what you said to mount.
> /proc/mounts shows what the current kernel state is.
> These may differ greatly.
>
> For all filesystems mounted by you using mount(8), a line is added
> to /etc/mtab, where the contents of that line is related to the
> given mount command, but not to what the kernel did.
>
> For the root filesystem, mount(8) writes an initial line in /etc/mtab
> taken from /etc/fstab. Again the information is from you, not from the kernel.
>
> >> # dmesg | grep 'Kernel command'
> >> Kernel command line: ro root=/dev/hda2 rootfstype=ext2
> > ...
> >> /dev/root / ext3 rw 0 0
>
> It would be a bad bug if the kernel mounted its root filesystem
> with a type different from the type given in "rootfstype=".
> But I see you use an initrd, and there can be all kinds of commands there.
--
Alberto Alonso Global Gate Systems LLC.
(512) 351-7233 http://www.ggsys.net
Hardware, consulting, sysadmin, monitoring and remote backups