2015-04-24 07:54:35

by Amir Goldstein

[permalink] [raw]
Subject: [RFC][PATCH] sysrq: Emergency Remount R/O in reverse order

Hi Al,

This patch is very naive. The fact that it tries to change a behavior
that pre-dates modern git history suggests that I may be missing something.

I am quite sure that the change of behavior is beneficial to my use case
which involves loop mounts on Android, hence uploaded for Android:
https://android-review.googlesource.com/148047

I am not sure about the implications for other use cases, hence this RFC.
Is there a reason to start remount from root fs, because later remounts
may hang or take too long before some watchdog kicks in?

Please enlighten me.
Thanks,

Amir.

---------- Forwarded message ----------


This change fixes a problem where reboot on Android panics the kernel
almost every time when file systems are mounted over loop devices.

Android reboot command does:
- sync
- echo u > /proc/sysrq-trigger
- syscall_reboot

The problem is with sysrq emergency remount R/O trying to remount-ro
in wrong order.
since /data is re-mounted ro before loop devices, loop device
remount-ro fails to flush the journal and panics the kernel:

EXT4-fs (loop0): Remounting filesystem read-only
EXT4-fs (loop0): previous I/O error to superblock detected
loop: Write error at byte offset 0, length 4096.
Buffer I/O error on device loop0, logical block 0
lost page write due to I/O error on loop0
Kernel panic - not syncing: EXT4-fs panic from previous error

The fix is quite simple. In do_emergency_remount(), use
list_for_each_entry_reverse() on sb list instead of list_for_each_entry().
It makes a lot of sense to umount the file systems in reverse order in
which they were added to sb list.

Signed-off-by: Amir Goldstein <[email protected]>
Acked-by: Oren Laadan <[email protected]>
---
fs/super.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/super.c b/fs/super.c
index 2b7dc90..f1315e0 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -774,7 +774,7 @@ static void do_emergency_remount(struct work_struct *work)
struct super_block *sb, *p = NULL;

spin_lock(&sb_lock);
- list_for_each_entry(sb, &super_blocks, s_list) {
+ list_for_each_entry_reverse(sb, &super_blocks, s_list) {
if (hlist_unhashed(&sb->s_instances))
continue;
sb->s_count++;
--
1.8.2


2015-04-29 18:34:47

by Amir Goldstein

[permalink] [raw]
Subject: Re: [RFC][PATCH] sysrq: Emergency Remount R/O in reverse order

Hi Ted,

Did you see my query about emergency remount of loop mounted ext4?
Do you have any insights to share?

Thanks,
Amir.

On Fri, Apr 24, 2015 at 10:54 AM, Amir Goldstein <[email protected]> wrote:
> Hi Al,
>
> This patch is very naive. The fact that it tries to change a behavior
> that pre-dates modern git history suggests that I may be missing something.
>
> I am quite sure that the change of behavior is beneficial to my use case
> which involves loop mounts on Android, hence uploaded for Android:
> https://android-review.googlesource.com/148047
>
> I am not sure about the implications for other use cases, hence this RFC.
> Is there a reason to start remount from root fs, because later remounts
> may hang or take too long before some watchdog kicks in?
>
> Please enlighten me.
> Thanks,
>
> Amir.
>
> ---------- Forwarded message ----------
>
>
> This change fixes a problem where reboot on Android panics the kernel
> almost every time when file systems are mounted over loop devices.
>
> Android reboot command does:
> - sync
> - echo u > /proc/sysrq-trigger
> - syscall_reboot
>
> The problem is with sysrq emergency remount R/O trying to remount-ro
> in wrong order.
> since /data is re-mounted ro before loop devices, loop device
> remount-ro fails to flush the journal and panics the kernel:
>
> EXT4-fs (loop0): Remounting filesystem read-only
> EXT4-fs (loop0): previous I/O error to superblock detected
> loop: Write error at byte offset 0, length 4096.
> Buffer I/O error on device loop0, logical block 0
> lost page write due to I/O error on loop0
> Kernel panic - not syncing: EXT4-fs panic from previous error
>
> The fix is quite simple. In do_emergency_remount(), use
> list_for_each_entry_reverse() on sb list instead of list_for_each_entry().
> It makes a lot of sense to umount the file systems in reverse order in
> which they were added to sb list.
>
> Signed-off-by: Amir Goldstein <[email protected]>
> Acked-by: Oren Laadan <[email protected]>
> ---
> fs/super.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/super.c b/fs/super.c
> index 2b7dc90..f1315e0 100644
> --- a/fs/super.c
> +++ b/fs/super.c
> @@ -774,7 +774,7 @@ static void do_emergency_remount(struct work_struct *work)
> struct super_block *sb, *p = NULL;
>
> spin_lock(&sb_lock);
> - list_for_each_entry(sb, &super_blocks, s_list) {
> + list_for_each_entry_reverse(sb, &super_blocks, s_list) {
> if (hlist_unhashed(&sb->s_instances))
> continue;
> sb->s_count++;
> --
> 1.8.2

2015-04-30 15:31:30

by Theodore Ts'o

[permalink] [raw]
Subject: Re: [RFC][PATCH] sysrq: Emergency Remount R/O in reverse order

On Wed, Apr 29, 2015 at 09:33:45PM +0300, Amir Goldstein wrote:
> Hi Ted,
>
> Did you see my query about emergency remount of loop mounted ext4?
> Do you have any insights to share?

Your patch makes sense to me. Doing the remount-ro in reverse order
that file systems were mounted sounds like the right thing to do.

I can imagine trying to make ext4 handle the situation where it is
trying to do an unmount, or remount r/o, and the device has gone
read-only in a similar situation as one where the device has
disappeared altogether, but it's not going to solve all possible
issues. Trying to unmount or remount r/o the most recently mounted
file system first is simpler and will solve more problems.

Al?

- Ted

> >
> > This change fixes a problem where reboot on Android panics the kernel
> > almost every time when file systems are mounted over loop devices.
> >
> > Android reboot command does:
> > - sync
> > - echo u > /proc/sysrq-trigger
> > - syscall_reboot
> >
> > The problem is with sysrq emergency remount R/O trying to remount-ro
> > in wrong order.
> > since /data is re-mounted ro before loop devices, loop device
> > remount-ro fails to flush the journal and panics the kernel:
> >
> > EXT4-fs (loop0): Remounting filesystem read-only
> > EXT4-fs (loop0): previous I/O error to superblock detected
> > loop: Write error at byte offset 0, length 4096.
> > Buffer I/O error on device loop0, logical block 0
> > lost page write due to I/O error on loop0
> > Kernel panic - not syncing: EXT4-fs panic from previous error
> >
> > The fix is quite simple. In do_emergency_remount(), use
> > list_for_each_entry_reverse() on sb list instead of list_for_each_entry().
> > It makes a lot of sense to umount the file systems in reverse order in
> > which they were added to sb list.
> >
> > Signed-off-by: Amir Goldstein <[email protected]>
> > Acked-by: Oren Laadan <[email protected]>
> > ---
> > fs/super.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/fs/super.c b/fs/super.c
> > index 2b7dc90..f1315e0 100644
> > --- a/fs/super.c
> > +++ b/fs/super.c
> > @@ -774,7 +774,7 @@ static void do_emergency_remount(struct work_struct
> > *work)
> > struct super_block *sb, *p = NULL;
> >
> > spin_lock(&sb_lock);
> > - list_for_each_entry(sb, &super_blocks, s_list) {
> > + list_for_each_entry_reverse(sb, &super_blocks, s_list) {
> > if (hlist_unhashed(&sb->s_instances))
> > continue;
> > sb->s_count++;
> > --
> > 1.8.2
> >

2015-05-07 06:38:48

by Amir Goldstein

[permalink] [raw]
Subject: Re: [RFC][PATCH] sysrq: Emergency Remount R/O in reverse order

On Thu, Apr 30, 2015 at 6:31 PM, Theodore Ts'o <[email protected]> wrote:
> On Wed, Apr 29, 2015 at 09:33:45PM +0300, Amir Goldstein wrote:
>> Hi Ted,
>>
>> Did you see my query about emergency remount of loop mounted ext4?
>> Do you have any insights to share?
>
> Your patch makes sense to me. Doing the remount-ro in reverse order
> that file systems were mounted sounds like the right thing to do.
>
> I can imagine trying to make ext4 handle the situation where it is
> trying to do an unmount, or remount r/o, and the device has gone
> read-only in a similar situation as one where the device has
> disappeared altogether, but it's not going to solve all possible
> issues. Trying to unmount or remount r/o the most recently mounted
> file system first is simpler and will solve more problems.
>
> Al?

Al,
Any comment?
Anybody else who dares to touch this code?
Anybody who benefits from this change?
Anybody who might be harmed from it?


>
> - Ted
>
>> >
>> > This change fixes a problem where reboot on Android panics the kernel
>> > almost every time when file systems are mounted over loop devices.
>> >
>> > Android reboot command does:
>> > - sync
>> > - echo u > /proc/sysrq-trigger
>> > - syscall_reboot
>> >
>> > The problem is with sysrq emergency remount R/O trying to remount-ro
>> > in wrong order.
>> > since /data is re-mounted ro before loop devices, loop device
>> > remount-ro fails to flush the journal and panics the kernel:
>> >
>> > EXT4-fs (loop0): Remounting filesystem read-only
>> > EXT4-fs (loop0): previous I/O error to superblock detected
>> > loop: Write error at byte offset 0, length 4096.
>> > Buffer I/O error on device loop0, logical block 0
>> > lost page write due to I/O error on loop0
>> > Kernel panic - not syncing: EXT4-fs panic from previous error
>> >
>> > The fix is quite simple. In do_emergency_remount(), use
>> > list_for_each_entry_reverse() on sb list instead of list_for_each_entry().
>> > It makes a lot of sense to umount the file systems in reverse order in
>> > which they were added to sb list.
>> >
>> > Signed-off-by: Amir Goldstein <[email protected]>
>> > Acked-by: Oren Laadan <[email protected]>
>> > ---
>> > fs/super.c | 2 +-
>> > 1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/fs/super.c b/fs/super.c
>> > index 2b7dc90..f1315e0 100644
>> > --- a/fs/super.c
>> > +++ b/fs/super.c
>> > @@ -774,7 +774,7 @@ static void do_emergency_remount(struct work_struct
>> > *work)
>> > struct super_block *sb, *p = NULL;
>> >
>> > spin_lock(&sb_lock);
>> > - list_for_each_entry(sb, &super_blocks, s_list) {
>> > + list_for_each_entry_reverse(sb, &super_blocks, s_list) {
>> > if (hlist_unhashed(&sb->s_instances))
>> > continue;
>> > sb->s_count++;
>> > --
>> > 1.8.2
>> >