2011-09-13 09:07:45

by Matt Parnell

[permalink] [raw]
Subject: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

Hi all... I usually run my system the zen kernel (zen-kernel.org), but
I've tried this with the default Archlinux kernel (pretty much a
vanilla build) as well and for some reason, the root drive won't mount
with the options I desire on anything above .29 The drive is an SSD,
and this is really getting on my nerves. I've looked at the
initscripts, fstab, everything else, and I just can't get any of this
to work properly. It was mounting with all the options I specified in
2.6.39, but since some change to the kernel (or something else?) I
can't mount with my options anymore. On top of this all, I've tried
using both /dev/root and /dev/sdb1, but for some reason the kernel's
only wanting to use /dev/root - I'm wondering if there's some kind of
conflict due to that symlink...

I know this isn't my initscripts, as I've also tried this using
openrc, aside from the default arch init system, and because upon
downgrading to 2.6.39 everything is working normally again, I know it
is the kernel. Information:

grub.cfg:

linux /boot/linux-zen root=/dev/sdb1 ro rootfstype=ext4
rootflags=data=writeback noatime barrier=0 discard noinitrd

fstab:

/dev/sdb1 / ext4 data=writeback,noatime,nodiratime,barrier=0,discard 0 1

Upon attempting to remount manually, I only get the following when I
try to use the options I specified above:

EXT4-fs (sdb1): Cannot change data mode on remount

Interestingly, this also happened and worked, which I think is a bug,
how can barrier=0 and 1 at the same time?

EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0

I'm currently using a .39 kernel and can live with it for now, but I
thought this was serious enough that the ext4 devs should hear about
it. If you need any more information, just let me know.

Thanks!

Matt Parnell


2011-09-13 10:25:52

by Christian Kujau

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On Tue, 13 Sep 2011 at 04:01, Matt Parnell wrote:
> I know this isn't my initscripts, as I've also tried this using
> openrc, aside from the default arch init system, and because upon
> downgrading to 2.6.39 everything is working normally again, I know it

Did you try booting with init=/bin/bash, i.e. bypassing all initscripts?
Also, try booting w/o an initrd image if you can, thus bypassing all the
early boot magic.

Shot in the dark: if it worked with 2.6.39 but not with 3.x and the
initscript have not changed, maybe some initscript went mad when it
could not parse the new kernel version number?

Christian.
--
BOFH excuse #141:

disks spinning backwards - toggle the hemisphere jumper.

2011-09-13 10:48:12

by Christian Kujau

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount


[ Cc'ing linux-ext4 again ]

On Tue, 13 Sep 2011 at 05:44, Matt Parnell wrote:
> > Did you try booting with init=/bin/bash, i.e. bypassing all initscripts?
> > Also, try booting w/o an initrd image if you can, thus bypassing all the
> > early boot magic.
>
> Not exactly, but I had it drop out to perform an fsck, and the mtab
> indicated it was mounted according to my options, but this was when
> the disk was mounted read only from grub. Also, I don't use an
> initrd...so that's not it either.

Yeah, mtab is no good when your rootfs is readonly. What did /proc/mounts
say?

C.

> > Shot in the dark: if it worked with 2.6.39 but not with 3.x and the
> > initscript have not changed, maybe some initscript went mad when it
> > could not parse the new kernel version number?
> >
> > Christian.
>
> It's not that either... Arch has been using 3.0+ as it's default for a
> few weeks now, so they would've updated their scripts for that.
> Furthermore, I tried using OpenRC (the gentoo init system) and got the
> same results.
>

--
BOFH excuse #359:

YOU HAVE AN I/O ERROR -> Incompetent Operator error

2011-09-13 14:49:47

by Eric Sandeen

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On 9/13/11 4:01 AM, Matt Parnell wrote:

> Interestingly, this also happened and worked, which I think is a bug,
> how can barrier=0 and 1 at the same time?
>
> EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0

Fairly certain that the parser will just take the last value specified
for the option.

/proc/mounts likely says "nobarrier" when you specify it this way?

-Eric

2011-10-09 22:10:55

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

FYI this behavior still occurs with every revision up to the current
3.0.6 working tree. I do not think it's my distro's initscripts,
they've been rolling 3.0.X kernels for some time and those behave the
same way, so it's not some sort of parsing issues - Arch's initscripts
are all bash based for the most part anyway, see the attached from my
system - that's the entire boot initscript minus my daemons.

There's no alternative init system for Archlinux at this point, I'm
working on porting OpenRC from gentoo as I have time to write
initscripts from scratch, but that's a slow moving project.

Whatever the case, the following are from my most recent attempt with
a 3.0.6 build. Manually attempting to remount rw still fails, and no
matter whether I use /dev/sdb1 or /dev/root, I get the same behavior.

/proc/mounts (as you can see it's not being remounted rw for some
reason - /dev/sdc1 is the flash drive I used to pipe this info onto):

rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4059824k,nr_inodes=1014956,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
/sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/run /run tmpfs rw,nosuid,nodev,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/sda4 /media/disk ext3
rw,nosuid,nodev,noatime,nodiratime,errors=continue,barrier=0,data=writeback
0 0
/dev/sdc1 /mnt/tmp vfat
rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
0 0

/etc/mtab (not useful, I know, but for reference):

/dev/root / ext4
rw,noatime,nodiratime,data=writeback,barrier=0,discard,commit=0 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4059656k,nr_inodes=1014914,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
/sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/run /run tmpfs rw,nosuid,nodev,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0


/etc/fstab:

#
# /etc/fstab: static file system information
#
devpts /dev/pts devpts defaults 0 0
shm /dev/shm tmpfs nodev,nosuid 0 0
/dev/sdb1 / ext4
defaults,noatime,nodiratime,data=writeback,barrier=0,discard 0 0
/dev/sda4 /media/disk ext3
defaults,data=writeback,noatime,nodiratime,user,exec 0 0
tmpfs /tmp tmpfs nodev,nosuid 0 0



>On 09/13/2011 09:49 AM, Eric Sandeen wrote:
>
>On 9/13/11 4:01 AM, Matt Parnell wrote:
>
>Interestingly, this also happened and worked, which I think is a bug,
>how can barrier=0 and 1 at the same time?
>
>EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0
>
>Fairly certain that the parser will just take the last value specified
>for the option.
>
>/proc/mounts likely says "nobarrier" when you specify it this way?
>
>-Eric


Attachments:
rc.sysinit (8.54 kB)

2011-10-09 23:44:34

by Theodore Ts'o

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>
> /proc/mounts (as you can see it's not being remounted rw for some
> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>
> rootfs / rootfs rw 0 0
> /dev/root / ext4 ro,relatime,barrier=1 0 0

Can you send me the output of /proc/cmdline? It looks like the
rootflags command line option isn't getting passed to the kernel for
some reason. That would explain why data=writeback isn't showing up
in /proc/mounts.

- Ted

2011-10-21 03:29:12

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

The output of /proc/cmdline on 3.0.7:

BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd


On 10/09/2011 06:44 PM, Ted Ts'o wrote:
> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>> /proc/mounts (as you can see it's not being remounted rw for some
>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>
>> rootfs / rootfs rw 0 0
>> /dev/root / ext4 ro,relatime,barrier=1 0 0
> Can you send me the output of /proc/cmdline? It looks like the
> rootflags command line option isn't getting passed to the kernel for
> some reason. That would explain why data=writeback isn't showing up
> in /proc/mounts.
>
> - Ted

2011-10-21 04:29:14

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

Also, note that the image is named vmlinuz26-zen there by mistake - I
fixed that in my build script, but even so, it makes no difference. In
3.0.7, I still can't remount root rw on ext4.

On 10/20/2011 10:29 PM, Matt Parnell wrote:
> The output of /proc/cmdline on 3.0.7:
>
> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
> rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd
>
>
> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
>> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>>> /proc/mounts (as you can see it's not being remounted rw for some
>>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>>
>>> rootfs / rootfs rw 0 0
>>> /dev/root / ext4 ro,relatime,barrier=1 0 0
>> Can you send me the output of /proc/cmdline? It looks like the
>> rootflags command line option isn't getting passed to the kernel for
>> some reason. That would explain why data=writeback isn't showing up
>> in /proc/mounts.
>>
>> - Ted

2011-10-22 07:49:16

by Theodore Ts'o

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On Thu, Oct 20, 2011 at 10:29:49PM -0500, Matt Parnell wrote:
> The output of /proc/cmdline on 3.0.7:
>
> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
> rootflags=data=writeback noatime nodiratime barrier=0 discard
> noinitrd
>
>
> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
> >On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
> >>/proc/mounts (as you can see it's not being remounted rw for some
> >>reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
> >>
> >>rootfs / rootfs rw 0 0
> >>/dev/root / ext4 ro,relatime,barrier=1 0 0

I haven't had a chance to check on v3.0 (I'm currently travelling and
penning this from an airplane), but on v3.1-rc3, it works for me under
KVM. I fire up kvm-qemu this way:

$QEMU -enable-kvm -boot order=c $NET \
-drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
-drive file=$VDB,cache=none,if=virtio \
-drive file=$VDC,cache=none,if=virtio \
-drive file=$VDD,cache=none,if=virtio \
-nographic -smp $NR_CPU -m $MEM \
--kernel $KERNEL \
--append "root=/dev/vda console=ttyS0,115200 rootflags=data=writeback" |\
tee $LOGFILE

with this in /proc/cmdline:

candygram:~# cat /proc/cmdline
root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint

... and this is what I have in /proc/mounts:

candygram:~# head -2 /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0

... and what I have in /etc/fstab:

UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02 / ext4 noatime,data=writeback 0 1

The kvm image I'm using is using Debian unstable image generated using
debootstrap, and uses no modules or initrd's --- which makes it easier
for me to boot since $KERNEL is set to:

/tyt/linux/ext4/arch/x86/boot/bzImage

This allows me to fire up test kernel right from my build tree,
without needing to set up any kind of initrd nonsense. :-)

- Ted

2011-10-22 07:51:39

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

That doesn't really help me at all, it's not rootflags=data=writeback
causing this, it's starting to make me think that arch's init may be to
blame, although I previously ruled it out...

This is all a big wtf though. I'm going to go complain at the arch devs
now... I've not finished porting openrc to it yet.

On 10/22/2011 12:00 AM, Ted Ts'o wrote:
> On Thu, Oct 20, 2011 at 10:29:49PM -0500, Matt Parnell wrote:
>> The output of /proc/cmdline on 3.0.7:
>>
>> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
>> rootflags=data=writeback noatime nodiratime barrier=0 discard
>> noinitrd
>>
>>
>> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
>>> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>>>> /proc/mounts (as you can see it's not being remounted rw for some
>>>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>>>
>>>> rootfs / rootfs rw 0 0
>>>> /dev/root / ext4 ro,relatime,barrier=1 0 0
> I haven't had a chance to check on v3.0 (I'm currently travelling and
> penning this from an airplane), but on v3.1-rc3, it works for me under
> KVM. I fire up kvm-qemu this way:
>
> $QEMU -enable-kvm -boot order=c $NET \
> -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
> -drive file=$VDB,cache=none,if=virtio \
> -drive file=$VDC,cache=none,if=virtio \
> -drive file=$VDD,cache=none,if=virtio \
> -nographic -smp $NR_CPU -m $MEM \
> --kernel $KERNEL \
> --append "root=/dev/vda console=ttyS0,115200 rootflags=data=writeback" |\
> tee $LOGFILE
>
> with this in /proc/cmdline:
>
> candygram:~# cat /proc/cmdline
> root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint
>
> ... and this is what I have in /proc/mounts:
>
> candygram:~# head -2 /proc/mounts
> rootfs / rootfs rw 0 0
> /dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0
>
> ... and what I have in /etc/fstab:
>
> UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02 / ext4 noatime,data=writeback 0 1
>
> The kvm image I'm using is using Debian unstable image generated using
> debootstrap, and uses no modules or initrd's --- which makes it easier
> for me to boot since $KERNEL is set to:
>
> /tyt/linux/ext4/arch/x86/boot/bzImage
>
> This allows me to fire up test kernel right from my build tree,
> without needing to set up any kind of initrd nonsense. :-)
>
> - Ted

2011-10-22 07:53:38

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

Also, I didn't intend to come across cross there... I appreciate the
efforts you are taking there.

It's just, I don't know what could've changed between .29 and 3.0 that
would break an entire init system so that it won't remount root rw...

On 10/22/2011 02:51 AM, Matt Parnell wrote:
> That doesn't really help me at all, it's not rootflags=data=writeback
> causing this, it's starting to make me think that arch's init may be
> to blame, although I previously ruled it out...
>
> This is all a big wtf though. I'm going to go complain at the arch
> devs now... I've not finished porting openrc to it yet.
>
> On 10/22/2011 12:00 AM, Ted Ts'o wrote:
>> On Thu, Oct 20, 2011 at 10:29:49PM -0500, Matt Parnell wrote:
>>> The output of /proc/cmdline on 3.0.7:
>>>
>>> BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4
>>> rootflags=data=writeback noatime nodiratime barrier=0 discard
>>> noinitrd
>>>
>>>
>>> On 10/09/2011 06:44 PM, Ted Ts'o wrote:
>>>> On Sun, Oct 09, 2011 at 05:10:54PM -0500, Matt Parnell wrote:
>>>>> /proc/mounts (as you can see it's not being remounted rw for some
>>>>> reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
>>>>>
>>>>> rootfs / rootfs rw 0 0
>>>>> /dev/root / ext4 ro,relatime,barrier=1 0 0
>> I haven't had a chance to check on v3.0 (I'm currently travelling and
>> penning this from an airplane), but on v3.1-rc3, it works for me under
>> KVM. I fire up kvm-qemu this way:
>>
>> $QEMU -enable-kvm -boot order=c $NET \
>> -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
>> -drive file=$VDB,cache=none,if=virtio \
>> -drive file=$VDC,cache=none,if=virtio \
>> -drive file=$VDD,cache=none,if=virtio \
>> -nographic -smp $NR_CPU -m $MEM \
>> --kernel $KERNEL \
>> --append "root=/dev/vda console=ttyS0,115200
>> rootflags=data=writeback" |\
>> tee $LOGFILE
>>
>> with this in /proc/cmdline:
>>
>> candygram:~# cat /proc/cmdline
>> root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint
>>
>> ... and this is what I have in /proc/mounts:
>>
>> candygram:~# head -2 /proc/mounts
>> rootfs / rootfs rw 0 0
>> /dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0
>>
>> ... and what I have in /etc/fstab:
>>
>> UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02 / ext4
>> noatime,data=writeback 0 1
>>
>> The kvm image I'm using is using Debian unstable image generated using
>> debootstrap, and uses no modules or initrd's --- which makes it easier
>> for me to boot since $KERNEL is set to:
>>
>> /tyt/linux/ext4/arch/x86/boot/bzImage
>>
>> This allows me to fire up test kernel right from my build tree,
>> without needing to set up any kind of initrd nonsense. :-)
>>
>> - Ted

2011-10-22 09:32:23

by Theodore Ts'o

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
> That doesn't really help me at all, it's not
> rootflags=data=writeback causing this, it's starting to make me
> think that arch's init may be to blame, although I previously ruled
> it out...

Well, it looks like rootflags=data=writeback is not making it to the
file system. That's why it's not showing up in /proc/mounts, from you
showed us. Can you look at the kernel dmesg?

You should see something like this:

[ 1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[ 1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[ 1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[ 1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.


The first line is the failure to mount the root file system as ext3.
The 2nd is the failure to mount the file system as ext2, using the
ext4 file system driver. The last two lines show the options show the
mount as ext4.

What do those two lines look to you. If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed
down to the file system. Then when you try to remount the file system
read/write, the fact that you have "data=writeback" in your /etc/fstab
causes the failure to remount.

If you simply remove that from /etc/fstab, things should work better.
The remount will preserve whatever data=journalling mode was in use
when the root file system was originallymounted as. If rootflags is
non-functional, then the file system won't be mounted as
data=writeback, but at least the boot sequence will continue without
blowing out.

- Ted

2011-11-07 02:18:40

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On 10/22/2011 02:32 AM, Ted Ts'o wrote:
> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>> That doesn't really help me at all, it's not
>> rootflags=data=writeback causing this, it's starting to make me
>> think that arch's init may be to blame, although I previously ruled
>> it out...
> Well, it looks like rootflags=data=writeback is not making it to the
> file system. That's why it's not showing up in /proc/mounts, from you
> showed us. Can you look at the kernel dmesg?
>
> You should see something like this:
>
> [ 1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
> [ 1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
> [ 1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
> [ 1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
>
>
> The first line is the failure to mount the root file system as ext3.
> The 2nd is the failure to mount the file system as ext2, using the
> ext4 file system driver. The last two lines show the options show the
> mount as ext4.
>
> What do those two lines look to you. If you don't see "Opts:
> data=writeback", then somehow the rootflags option isn't getting passed
> down to the file system. Then when you try to remount the file system
> read/write, the fact that you have "data=writeback" in your /etc/fstab
> causes the failure to remount.
>
> If you simply remove that from /etc/fstab, things should work better.
> The remount will preserve whatever data=journalling mode was in use
> when the root file system was originallymounted as. If rootflags is
> non-functional, then the file system won't be mounted as
> data=writeback, but at least the boot sequence will continue without
> blowing out.
>
> - Ted

Since then, I created a fresh partition and re rsynced my root over onto
it after making it WITHOUT a journal using the current sysrescue64 cd
(kernel 3.0.1).

Same thing is happening, and the drive still refuses to mount any other
way except

/dev/root / ext4 rw,relatime,user_xattr,acl,barrier=1 0 0

Despite my specifying these in fstab and grub.cfg ...it's like it's not
even passing the options somewhere, either from grub2 to the kernel or
from my initscripts...but since I've tried using openrc too and get this
result, I'm not sure what to do. Note that using both /dev/sdb1 and
/dev/root don't interchange between the fstab and the kernel...I think
the /dev/root symlink may be breaking my initscripts.

/dev/sdb1 / ext4
noatime,nodiratime,data=writeback,nouser_xattr,barrier=0,commit=10 0 0

2011-11-07 02:24:46

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On 10/22/2011 02:32 AM, Ted Ts'o wrote:
> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>> That doesn't really help me at all, it's not
>> rootflags=data=writeback causing this, it's starting to make me
>> think that arch's init may be to blame, although I previously ruled
>> it out...
> Well, it looks like rootflags=data=writeback is not making it to the
> file system. That's why it's not showing up in /proc/mounts, from you
> showed us. Can you look at the kernel dmesg?
>
> You should see something like this:
>
> [ 1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
> [ 1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
> [ 1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
> [ 1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
>
>
> The first line is the failure to mount the root file system as ext3.
> The 2nd is the failure to mount the file system as ext2, using the
> ext4 file system driver. The last two lines show the options show the
> mount as ext4.
>
> What do those two lines look to you. If you don't see "Opts:
> data=writeback", then somehow the rootflags option isn't getting passed
> down to the file system. Then when you try to remount the file system
> read/write, the fact that you have "data=writeback" in your /etc/fstab
> causes the failure to remount.
>
> If you simply remove that from /etc/fstab, things should work better.
> The remount will preserve whatever data=journalling mode was in use
> when the root file system was originallymounted as. If rootflags is
> non-functional, then the file system won't be mounted as
> data=writeback, but at least the boot sequence will continue without
> blowing out.
>
> - Ted
Looks like the kernel isn't creating the /dev/root link/block device,
either all of the time or some of the time, I'm confused.

2011-11-07 03:27:49

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On 11/06/2011 06:24 PM, Matt Parnell wrote:
> On 10/22/2011 02:32 AM, Ted Ts'o wrote:
>> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>>> That doesn't really help me at all, it's not
>>> rootflags=data=writeback causing this, it's starting to make me
>>> think that arch's init may be to blame, although I previously ruled
>>> it out...
>> Well, it looks like rootflags=data=writeback is not making it to the
>> file system. That's why it's not showing up in /proc/mounts, from you
>> showed us. Can you look at the kernel dmesg?
>>
>> You should see something like this:
>>
>> [ 1.421146] EXT3-fs (vda): error: couldn't mount because of
>> unsupported optional features (240)
>> [ 1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature
>> incompatibilities
>> [ 1.454631] EXT4-fs (vda): mounted filesystem with writeback data
>> mode. Opts: data=writeback
>> [ 1.455966] VFS: Mounted root (ext4 filesystem) readonly on device
>> 254:0.
>>
>>
>> The first line is the failure to mount the root file system as ext3.
>> The 2nd is the failure to mount the file system as ext2, using the
>> ext4 file system driver. The last two lines show the options show the
>> mount as ext4.
>>
>> What do those two lines look to you. If you don't see "Opts:
>> data=writeback", then somehow the rootflags option isn't getting passed
>> down to the file system. Then when you try to remount the file system
>> read/write, the fact that you have "data=writeback" in your /etc/fstab
>> causes the failure to remount.
>>
>> If you simply remove that from /etc/fstab, things should work better.
>> The remount will preserve whatever data=journalling mode was in use
>> when the root file system was originallymounted as. If rootflags is
>> non-functional, then the file system won't be mounted as
>> data=writeback, but at least the boot sequence will continue without
>> blowing out.
>>
>> - Ted
> Looks like the kernel isn't creating the /dev/root link/block device,
> either all of the time or some of the time, I'm confused.
I figured it out. data=writeback isn't needed and is done by default if
there's no journal, as per commit
373cd5c53d5ea6622c319ecd84e29e2737d488bd
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd

...sorry about throwing you guys through loops over this, although I'm
guessing there's still a bug in that the kernel should just warn the
user about the option when it's used and otherwise ignore it instead of
breaking if there's no journal.

2011-11-07 03:33:14

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On 11/06/2011 09:26 PM, Matt Parnell wrote:
> On 11/06/2011 06:24 PM, Matt Parnell wrote:
>> On 10/22/2011 02:32 AM, Ted Ts'o wrote:
>>> On Sat, Oct 22, 2011 at 02:51:35AM -0500, Matt Parnell wrote:
>>>> That doesn't really help me at all, it's not
>>>> rootflags=data=writeback causing this, it's starting to make me
>>>> think that arch's init may be to blame, although I previously ruled
>>>> it out...
>>> Well, it looks like rootflags=data=writeback is not making it to the
>>> file system. That's why it's not showing up in /proc/mounts, from you
>>> showed us. Can you look at the kernel dmesg?
>>>
>>> You should see something like this:
>>>
>>> [ 1.421146] EXT3-fs (vda): error: couldn't mount because of
>>> unsupported optional features (240)
>>> [ 1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature
>>> incompatibilities
>>> [ 1.454631] EXT4-fs (vda): mounted filesystem with writeback data
>>> mode. Opts: data=writeback
>>> [ 1.455966] VFS: Mounted root (ext4 filesystem) readonly on
>>> device 254:0.
>>>
>>>
>>> The first line is the failure to mount the root file system as ext3.
>>> The 2nd is the failure to mount the file system as ext2, using the
>>> ext4 file system driver. The last two lines show the options show the
>>> mount as ext4.
>>>
>>> What do those two lines look to you. If you don't see "Opts:
>>> data=writeback", then somehow the rootflags option isn't getting passed
>>> down to the file system. Then when you try to remount the file system
>>> read/write, the fact that you have "data=writeback" in your /etc/fstab
>>> causes the failure to remount.
>>>
>>> If you simply remove that from /etc/fstab, things should work better.
>>> The remount will preserve whatever data=journalling mode was in use
>>> when the root file system was originallymounted as. If rootflags is
>>> non-functional, then the file system won't be mounted as
>>> data=writeback, but at least the boot sequence will continue without
>>> blowing out.
>>>
>>> - Ted
>> Looks like the kernel isn't creating the /dev/root link/block device,
>> either all of the time or some of the time, I'm confused.
> I figured it out. data=writeback isn't needed and is done by default
> if there's no journal, as per commit
> 373cd5c53d5ea6622c319ecd84e29e2737d488bd
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
>
> ...sorry about throwing you guys through loops over this, although I'm
> guessing there's still a bug in that the kernel should just warn the
> user about the option when it's used and otherwise ignore it instead
> of breaking if there's no journal.
That is, you were right in removing rootflags=data=writeback from the
kernel boot params, and removing data=writeback from fstab would fix it
- that commit is where things break, though...

So really instead of breaking it should log that it's ignoring that flag
becuase it's enabled by default when there's no journal.

2011-11-07 15:48:03

by Theodore Ts'o

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On Sun, Nov 06, 2011 at 09:26:57PM -0800, Matt Parnell wrote:
> I figured it out. data=writeback isn't needed and is done by default
> if there's no journal, as per commit
> 373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
>
> ...sorry about throwing you guys through loops over this, although
> I'm guessing there's still a bug in that the kernel should just warn
> the user about the option when it's used and otherwise ignore it
> instead of breaking if there's no journal.

Ah, OK. It wasn't clear to me that you were running without a year.
You're right, we shouldn't complain in that case, since we ignore the
messages when mounting the file system. Actually, we parse the
options when we are mounting the file system, but then we zero them
out when we discover that there's no journal; but in any case, we
should similarly ignore the pointless data=* mount options (for a file
system w/o a journal) on the remount operation as well.

- Ted

commit eb513689c97e3e73bb9b4459d490a8e894b4a546
Author: Theodore Ts'o <[email protected]>
Date: Mon Nov 7 10:47:42 2011 -0500

ext4: ignore journalled data options on remount if fs has no journal

This avoids a confusing failure in the init scripts when the
/etc/fstab has data=writeback or data=journal but the file system does
not have a journal. So check for this case explicitly, and warn the
user that we are ignoring the (pointless, since they have no journal)
data=* mount option.

Signed-off-by: "Theodore Ts'o" <[email protected]>

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 9953d80..0435013 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -1683,7 +1683,9 @@ static int parse_options(char *options, struct super_block *sb,
data_opt = EXT4_MOUNT_WRITEBACK_DATA;
datacheck:
if (is_remount) {
- if (test_opt(sb, DATA_FLAGS) != data_opt) {
+ if (!sbi->s_journal)
+ ext4_msg(sb, KERN_WARNING, "Remounting file system with no journal so ignoring journalled data option");
+ else if (test_opt(sb, DATA_FLAGS) != data_opt) {
ext4_msg(sb, KERN_ERR,
"Cannot change data mode on remount");
return 0;

2011-11-07 23:49:10

by Matt Parnell

[permalink] [raw]
Subject: Re: Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

On 11/07/2011 07:47 AM, Ted Ts'o wrote:
> On Sun, Nov 06, 2011 at 09:26:57PM -0800, Matt Parnell wrote:
>> I figured it out. data=writeback isn't needed and is done by default
>> if there's no journal, as per commit
>> 373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
>>
>> ...sorry about throwing you guys through loops over this, although
>> I'm guessing there's still a bug in that the kernel should just warn
>> the user about the option when it's used and otherwise ignore it
>> instead of breaking if there's no journal.
> Ah, OK. It wasn't clear to me that you were running without a year.
> You're right, we shouldn't complain in that case, since we ignore the
> messages when mounting the file system. Actually, we parse the
> options when we are mounting the file system, but then we zero them
> out when we discover that there's no journal; but in any case, we
> should similarly ignore the pointless data=* mount options (for a file
> system w/o a journal) on the remount operation as well.
>
> - Ted
>
> commit eb513689c97e3e73bb9b4459d490a8e894b4a546
> Author: Theodore Ts'o<[email protected]>
> Date: Mon Nov 7 10:47:42 2011 -0500
>
> ext4: ignore journalled data options on remount if fs has no journal
>
> This avoids a confusing failure in the init scripts when the
> /etc/fstab has data=writeback or data=journal but the file system does
> not have a journal. So check for this case explicitly, and warn the
> user that we are ignoring the (pointless, since they have no journal)
> data=* mount option.
>
> Signed-off-by: "Theodore Ts'o"<[email protected]>
>
> diff --git a/fs/ext4/super.c b/fs/ext4/super.c
> index 9953d80..0435013 100644
> --- a/fs/ext4/super.c
> +++ b/fs/ext4/super.c
> @@ -1683,7 +1683,9 @@ static int parse_options(char *options, struct super_block *sb,
> data_opt = EXT4_MOUNT_WRITEBACK_DATA;
> datacheck:
> if (is_remount) {
> - if (test_opt(sb, DATA_FLAGS) != data_opt) {
> + if (!sbi->s_journal)
> + ext4_msg(sb, KERN_WARNING, "Remounting file system with no journal so ignoring journalled data option");
> + else if (test_opt(sb, DATA_FLAGS) != data_opt) {
> ext4_msg(sb, KERN_ERR,
> "Cannot change data mode on remount");
> return 0;
Great! I'm glad we managed to track this one down. I'm not using the
flags anymore, but I'll be glad when your patch there makes it into ext4
for other users.