2024-03-20 00:14:53

by Jaegeuk Kim

Subject: [PATCH] f2fs: avoid the deadlock case when stopping discard thread

f2fs_ioc_shutdown(F2FS_GOING_DOWN_NOSYNC) issue_discard_thread
- mnt_want_write_file()
- sb_start_write(SB_FREEZE_WRITE)
- sb_start_intwrite(SB_FREEZE_FS);
- f2fs_stop_checkpoint(sbi, false, : waiting
STOP_CP_REASON_SHUTDOWN);
- f2fs_stop_discard_thread(sbi);
- kthread_stop()
: waiting

- mnt_drop_write_file(filp);

Signed-off-by: Jaegeuk Kim <[email protected]>
---
fs/f2fs/segment.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index 4fd76e867e0a..088b8c48cffa 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -1923,7 +1923,9 @@ static int issue_discard_thread(void *data)
continue;
}

- sb_start_intwrite(sbi->sb);
+ /* Avoid the deadlock from F2FS_GOING_DOWN_NOSYNC. */
+ if (!sb_start_intwrite_trylock(sbi->sb))
+ continue;

issued = __issue_discard_cmd(sbi, &dpolicy);
if (issued > 0) {
--
2.44.0.291.gc1ea87d7ee-goog

2024-03-20 15:32:56

by Chao Yu

Subject: Re: [f2fs-dev] [PATCH] f2fs: avoid the deadlock case when stopping discard thread

On 2024/3/20 8:14, Jaegeuk Kim wrote:
> f2fs_ioc_shutdown(F2FS_GOING_DOWN_NOSYNC) issue_discard_thread
> - mnt_want_write_file()
> - sb_start_write(SB_FREEZE_WRITE)
> - sb_start_intwrite(SB_FREEZE_FS);
> - f2fs_stop_checkpoint(sbi, false, : waiting
> STOP_CP_REASON_SHUTDOWN);
> - f2fs_stop_discard_thread(sbi);
> - kthread_stop()
> : waiting
>
> - mnt_drop_write_file(filp);
>
> Signed-off-by: Jaegeuk Kim <[email protected]>

Reviewed-by: Chao Yu <[email protected]>

Thanks,

2024-03-21 22:43:28

by Hillf Danton

Subject: Re: [PATCH] f2fs: avoid the deadlock case when stopping discard thread

On Tue, 19 Mar 2024 17:14:42 -0700 Jaegeuk Kim <[email protected]>
> f2fs_ioc_shutdown(F2FS_GOING_DOWN_NOSYNC) issue_discard_thread
> - mnt_want_write_file()
> - sb_start_write(SB_FREEZE_WRITE)
__sb_start_write()
percpu_down_read()
> - sb_start_intwrite(SB_FREEZE_FS);
__sb_start_write()
percpu_down_read()

Given lock acquirers for read on both sides, wtf deadlock are you fixing?

> - f2fs_stop_checkpoint(sbi, false, : waiting
> STOP_CP_REASON_SHUTDOWN);
> - f2fs_stop_discard_thread(sbi);
> - kthread_stop()
> : waiting
>
> - mnt_drop_write_file(filp);

More important, feel free to add in spin.

Reported-by: "Light Hsieh (謝明燈)" <[email protected]>

2024-03-22 00:29:13

by Jaegeuk Kim

Subject: Re: [PATCH] f2fs: avoid the deadlock case when stopping discard thread

On 03/22, Hillf Danton wrote:
> On Tue, 19 Mar 2024 17:14:42 -0700 Jaegeuk Kim <[email protected]>
> > f2fs_ioc_shutdown(F2FS_GOING_DOWN_NOSYNC) issue_discard_thread
> > - mnt_want_write_file()
> > - sb_start_write(SB_FREEZE_WRITE)
> __sb_start_write()
> percpu_down_read()
> > - sb_start_intwrite(SB_FREEZE_FS);
> __sb_start_write()
> percpu_down_read()
>
> Given lock acquirers for read on both sides, wtf deadlock are you fixing?

Damn. I couldn't think _write uses _read sem.

>
> > - f2fs_stop_checkpoint(sbi, false, : waiting
> > STOP_CP_REASON_SHUTDOWN);
> > - f2fs_stop_discard_thread(sbi);
> > - kthread_stop()
> > : waiting
> >
> > - mnt_drop_write_file(filp);
>
> More important, feel free to add in spin.

I posted this patch before Light reported.

And, in the report, I didn't get this:

f2fs_ioc_shutdown() --> freeze_bdev() --> freeze_super() --> sb_wait_write(sb, SB_FREEZE_FS) --> ... ->percpu_down_write().

because f2fs_ioc_shutdown() calls f2fs_stop_discard_thread() after thaw_bdev()
like this order.

-> freeze_bdev()
-> thaw_bdev()
-> f2fs_stop_discard_thread()

Am I missing something?

>
> Reported-by: "Light Hsieh (謝明燈)" <[email protected]>

2024-03-22 11:34:41

by Hillf Danton

Subject: Re: [PATCH] f2fs: avoid the deadlock case when stopping discard thread

On Thu, 21 Mar 2024 17:29:03 -0700 Jaegeuk Kim <[email protected]>
>
> I posted this patch before Light reported.

Yeah, his report's timestamp is 2024-03-20 6:59, nearly 7 hours later,
which shows that you constructed the deadlock with nothing to do with
his report.
>
> And, in the report, I didn't get this:
>
> f2fs_ioc_shutdown() --> freeze_bdev() --> freeze_super() --> sb_wait_write(sb, SB_FREEZE_FS) --> ... ->percpu_down_write().
>
> because f2fs_ioc_shutdown() calls f2fs_stop_discard_thread() after thaw_bdev()
> like this order.
>
> -> freeze_bdev()
> -> thaw_bdev()
> -> f2fs_stop_discard_thread()
>
> Am I missing something?

Light, could you specify to help Jaegeuk understand the deadlock you reported?