2024-03-20 06:59:41

by Light Hsieh (謝明燈)

[permalink] [raw]
Subject: f2fs F2FS_IOC_SHUTDOWN hang issue

Hi Jaegeuk:

We encounter a deadlock issue when Android is going to poweroff.
Please help check.

When unmounting of  f2fs partition fail in Android poweroff procedure, init thread (pid = 1) invoke F2FS_IOC_SHUTDOWN  ioctl with arg F2FS_GOING_DOWN_FULLSYNC.
This ioctl cause down_write of a semaphore in the following call sequence:
        f2fs_ioc_shutdown() --> freeze_bdev() --> freeze_super() --> sb_wait_write(sb, SB_FREEZE_FS) --> ... ->percpu_down_write().

f2fs_ioc_shutdown() will later invoke f2fs_stop_discard_thread() and wait for stopping of f2fs_discard thread in the following call sequence:
        f2fs_ioc_shutdown() -->f2fs_stop_discard_thread() -->kthread_stop(discard_thread) --> wait_for_completion().
That is, init thread go sleep with a write semaphore.

f2fs_discard thread is then waken up to process f2fs discard.
However, f2fs_discard threshold may then hang because failing to get the semaphore aleady obtained by the slept init thread: 
        issue_discard_thread() --> sb_start_intwrite() -->sb_start_write(sb, SB_FREEZE_FS) --> percpu_down_read()

Light


2024-03-20 20:06:41

by Jaegeuk Kim

[permalink] [raw]
Subject: Re: f2fs F2FS_IOC_SHUTDOWN hang issue

Can you try this?

https://patchwork.kernel.org/project/f2fs/patch/[email protected]/

On 03/20, Light Hsieh (謝明燈) wrote:
> Hi Jaegeuk:
>
> We encounter a deadlock issue when Android is going to poweroff.
> Please help check.
>
> When unmounting of f2fs partition fail in Android poweroff procedure, init thread (pid = 1) invoke F2FS_IOC_SHUTDOWN ioctl with arg F2FS_GOING_DOWN_FULLSYNC.
> This ioctl cause down_write of a semaphore in the following call sequence:
> f2fs_ioc_shutdown() --> freeze_bdev() --> freeze_super() --> sb_wait_write(sb, SB_FREEZE_FS) --> ... ->percpu_down_write().
>
> f2fs_ioc_shutdown() will later invoke f2fs_stop_discard_thread() and wait for stopping of f2fs_discard thread in the following call sequence:
> f2fs_ioc_shutdown() -->f2fs_stop_discard_thread() -->kthread_stop(discard_thread) --> wait_for_completion().
> That is, init thread go sleep with a write semaphore.
>
> f2fs_discard thread is then waken up to process f2fs discard.
> However, f2fs_discard threshold may then hang because failing to get the semaphore aleady obtained by the slept init thread:
> issue_discard_thread() --> sb_start_intwrite() -->sb_start_write(sb, SB_FREEZE_FS) --> percpu_down_read()
>
> Light

2024-03-20 23:35:17

by Light Hsieh (謝明燈)

[permalink] [raw]
Subject: 回覆: f2fs F2FS_IOC_SHUTDOWN hang issue

On 2024/3/20 8:14, Jaegeuk Kim wrote:
> f2fs_ioc_shutdown(F2FS_GOING_DOWN_NOSYNC)  issue_discard_thread
>   - mnt_want_write_file()
>     - sb_start_write(SB_FREEZE_WRITE)
>                                               - sb_start_intwrite(SB_FREEZE_FS);
>   - f2fs_stop_checkpoint(sbi, false,            : waiting
>      STOP_CP_REASON_SHUTDOWN);
>   - f2fs_stop_discard_thread(sbi);
>     - kthread_stop()
>       : waiting
>
>   - mnt_drop_write_file(filp);
>
> Signed-off-by: Jaegeuk Kim <[email protected]>

The case I encounter is f2fs_ic_shutdown with arg  F2FS_GOING_DOWN_FULLSYNC, not  F2FS_GOING_DOWN_NOSYNC.

Or you are meaning that: besides the kernel patch, I need to change the invoked F2FS_IOC_SHUTDOWN to use arg F2FS_GOING_DOWN_NOSYNC?



2024-03-21 00:39:32

by Jaegeuk Kim

[permalink] [raw]
Subject: Re: 回覆 : f2fs F2FS_IOC_SHUTDOWN hang issue

On 03/20, Light Hsieh (謝明燈) wrote:
> On 2024/3/20 8:14, Jaegeuk Kim wrote:
> > f2fs_ioc_shutdown(F2FS_GOING_DOWN_NOSYNC)  issue_discard_thread
> >   - mnt_want_write_file()
> >     - sb_start_write(SB_FREEZE_WRITE)
> >                                               - sb_start_intwrite(SB_FREEZE_FS);
> >   - f2fs_stop_checkpoint(sbi, false,            : waiting
> >      STOP_CP_REASON_SHUTDOWN);
> >   - f2fs_stop_discard_thread(sbi);
> >     - kthread_stop()
> >       : waiting
> >
> >   - mnt_drop_write_file(filp);
> >
> > Signed-off-by: Jaegeuk Kim <[email protected]>
>
> The case I encounter is f2fs_ic_shutdown with arg  F2FS_GOING_DOWN_FULLSYNC, not  F2FS_GOING_DOWN_NOSYNC.
>
> Or you are meaning that: besides the kernel patch, I need to change the invoked F2FS_IOC_SHUTDOWN to use arg F2FS_GOING_DOWN_NOSYNC?

I think this patch also addresses your case by using trylock.

>
>
>