From: Daeho Jeong <[email protected]>
We detected the below circular locking dependency between sb_internal
and fs_reclaim. So, removed it by calling dquot_initialize() before
sb_start_intwrite().
======================================================
WARNING: possible circular locking dependency detected
------------------------------------------------------
kswapd0/133 is trying to acquire lock:
ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
but task is already holding lock:
ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
__fs_reclaim_acquire+0x4/0x50
which lock already depends on the new lock.
...
other info that might help us debug this:
Chain exists of:
sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(fs_reclaim);
lock(&s->s_dquot.dqio_sem);
lock(fs_reclaim);
lock(sb_internal#2);
Signed-off-by: Daeho Jeong <[email protected]>
---
fs/f2fs/super.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 86eeb019cc52..a133932333c5 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
/* should remain fi->extent_tree for writepage */
f2fs_destroy_extent_node(inode);
+ dquot_initialize(inode);
+
sb_start_intwrite(inode->i_sb);
f2fs_i_size_write(inode, 0);
--
2.33.0.1079.g6e70778dc9-goog
On 2021/10/15 3:05, Daeho Jeong wrote:
> From: Daeho Jeong <[email protected]>
>
> We detected the below circular locking dependency between sb_internal
> and fs_reclaim. So, removed it by calling dquot_initialize() before
> sb_start_intwrite().
>
> ======================================================
> WARNING: possible circular locking dependency detected
> ------------------------------------------------------
> kswapd0/133 is trying to acquire lock:
> ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
>
> but task is already holding lock:
> ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
> __fs_reclaim_acquire+0x4/0x50
>
> which lock already depends on the new lock.
> ...
> other info that might help us debug this:
>
> Chain exists of:
>
> sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(fs_reclaim);
> lock(&s->s_dquot.dqio_sem);
> lock(fs_reclaim);
> lock(sb_internal#2);
Sorry, I still didn't get the root cause of this deadlock issue, could
you please explain more about this?
And why calling dquot_initialize() in drop_inode() could break the
circular locking dependency?
Thanks,
>
> Signed-off-by: Daeho Jeong <[email protected]>
> ---
> fs/f2fs/super.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> index 86eeb019cc52..a133932333c5 100644
> --- a/fs/f2fs/super.c
> +++ b/fs/f2fs/super.c
> @@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
> /* should remain fi->extent_tree for writepage */
> f2fs_destroy_extent_node(inode);
>
> + dquot_initialize(inode);
> +
> sb_start_intwrite(inode->i_sb);
> f2fs_i_size_write(inode, 0);
>
>
There is a deadlock between sb_internal lock (sb_start_intwrite()) and
dquot related lock.
It's because we call f2fs_truncate(), which eventually calls
dquot_initialize(), while holding sb_internal lock.
So, I called dquot_initialize() in advance to make the 2nd calling of
it in f2fs_truncate() ineffective.
This is similar with the thing in f2fs_evict_inode() in inode.c
Thanks,
On Thu, Oct 21, 2021 at 5:11 AM Chao Yu <[email protected]> wrote:
>
> On 2021/10/15 3:05, Daeho Jeong wrote:
> > From: Daeho Jeong <[email protected]>
> >
> > We detected the below circular locking dependency between sb_internal
> > and fs_reclaim. So, removed it by calling dquot_initialize() before
> > sb_start_intwrite().
> >
> > ======================================================
> > WARNING: possible circular locking dependency detected
> > ------------------------------------------------------
> > kswapd0/133 is trying to acquire lock:
> > ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
> >
> > but task is already holding lock:
> > ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
> > __fs_reclaim_acquire+0x4/0x50
> >
> > which lock already depends on the new lock.
> > ...
> > other info that might help us debug this:
> >
> > Chain exists of:
> >
> > sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
> >
> > Possible unsafe locking scenario:
> >
> > CPU0 CPU1
> > ---- ----
> > lock(fs_reclaim);
> > lock(&s->s_dquot.dqio_sem);
> > lock(fs_reclaim);
> > lock(sb_internal#2);
>
> Sorry, I still didn't get the root cause of this deadlock issue, could
> you please explain more about this?
>
> And why calling dquot_initialize() in drop_inode() could break the
> circular locking dependency?
>
> Thanks,
>
> >
> > Signed-off-by: Daeho Jeong <[email protected]>
> > ---
> > fs/f2fs/super.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> > index 86eeb019cc52..a133932333c5 100644
> > --- a/fs/f2fs/super.c
> > +++ b/fs/f2fs/super.c
> > @@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
> > /* should remain fi->extent_tree for writepage */
> > f2fs_destroy_extent_node(inode);
> >
> > + dquot_initialize(inode);
> > +
> > sb_start_intwrite(inode->i_sb);
> > f2fs_i_size_write(inode, 0);
> >
> >
On 2021/10/22 0:44, Daeho Jeong wrote:
> There is a deadlock between sb_internal lock (sb_start_intwrite()) and
> dquot related lock.
> It's because we call f2fs_truncate(), which eventually calls
> dquot_initialize(), while holding sb_internal lock.
> So, I called dquot_initialize() in advance to make the 2nd calling of
> it in f2fs_truncate() ineffective.
> This is similar with the thing in f2fs_evict_inode() in inode.c
Well, if dquot_initialize() fails in f2fs_drop_inode(), will we still run
into deadlock?
Thanks,
>
> Thanks,
>
> On Thu, Oct 21, 2021 at 5:11 AM Chao Yu <[email protected]> wrote:
>>
>> On 2021/10/15 3:05, Daeho Jeong wrote:
>>> From: Daeho Jeong <[email protected]>
>>>
>>> We detected the below circular locking dependency between sb_internal
>>> and fs_reclaim. So, removed it by calling dquot_initialize() before
>>> sb_start_intwrite().
>>>
>>> ======================================================
>>> WARNING: possible circular locking dependency detected
>>> ------------------------------------------------------
>>> kswapd0/133 is trying to acquire lock:
>>> ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
>>>
>>> but task is already holding lock:
>>> ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
>>> __fs_reclaim_acquire+0x4/0x50
>>>
>>> which lock already depends on the new lock.
>>> ...
>>> other info that might help us debug this:
>>>
>>> Chain exists of:
>>>
>>> sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
>>>
>>> Possible unsafe locking scenario:
>>>
>>> CPU0 CPU1
>>> ---- ----
>>> lock(fs_reclaim);
>>> lock(&s->s_dquot.dqio_sem);
>>> lock(fs_reclaim);
>>> lock(sb_internal#2);
>>
>> Sorry, I still didn't get the root cause of this deadlock issue, could
>> you please explain more about this?
>>
>> And why calling dquot_initialize() in drop_inode() could break the
>> circular locking dependency?
>>
>> Thanks,
>>
>>>
>>> Signed-off-by: Daeho Jeong <[email protected]>
>>> ---
>>> fs/f2fs/super.c | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
>>> index 86eeb019cc52..a133932333c5 100644
>>> --- a/fs/f2fs/super.c
>>> +++ b/fs/f2fs/super.c
>>> @@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
>>> /* should remain fi->extent_tree for writepage */
>>> f2fs_destroy_extent_node(inode);
>>>
>>> + dquot_initialize(inode);
>>> +
>>> sb_start_intwrite(inode->i_sb);
>>> f2fs_i_size_write(inode, 0);
>>>
>>>
On Fri, Oct 22, 2021 at 8:32 AM Chao Yu <[email protected]> wrote:
>
> On 2021/10/22 0:44, Daeho Jeong wrote:
> > There is a deadlock between sb_internal lock (sb_start_intwrite()) and
> > dquot related lock.
> > It's because we call f2fs_truncate(), which eventually calls
> > dquot_initialize(), while holding sb_internal lock.
> > So, I called dquot_initialize() in advance to make the 2nd calling of
> > it in f2fs_truncate() ineffective.
> > This is similar with the thing in f2fs_evict_inode() in inode.c
>
> Well, if dquot_initialize() fails in f2fs_drop_inode(), will we still run
> into deadlock?
>
Do you think the same issue is in f2fs_evict_inode() in inode.c?
In fact, I picked up the idea from here.
err = dquot_initialize(inode);
if (err) {
err = 0;
set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
}
f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
sb_start_intwrite(inode->i_sb);
set_inode_flag(inode, FI_NO_ALLOC);
i_size_write(inode, 0);
retry:
if (F2FS_HAS_BLOCKS(inode))
err = f2fs_truncate(inode);
On 2021/10/26 0:22, Daeho Jeong wrote:
> On Fri, Oct 22, 2021 at 8:32 AM Chao Yu <[email protected]> wrote:
>>
>> On 2021/10/22 0:44, Daeho Jeong wrote:
>>> There is a deadlock between sb_internal lock (sb_start_intwrite()) and
>>> dquot related lock.
>>> It's because we call f2fs_truncate(), which eventually calls
>>> dquot_initialize(), while holding sb_internal lock.
>>> So, I called dquot_initialize() in advance to make the 2nd calling of
>>> it in f2fs_truncate() ineffective.
>>> This is similar with the thing in f2fs_evict_inode() in inode.c
>>
>> Well, if dquot_initialize() fails in f2fs_drop_inode(), will we still run
>> into deadlock?
>>
>
> Do you think the same issue is in f2fs_evict_inode() in inode.c?
Yes, I doubt the problem may also happen in f2fs_evict_inode() with below
callpath:
- evict_inode
- dquot_initialize failed
- sb_start_intwrite
- f2fs_truncate
- dquot_initialize lock dqio_sem
How about this?
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
old mode 100644
new mode 100755
index b24b9bc..0e49593
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -728,6 +728,7 @@ enum {
FI_ENABLE_COMPRESS, /* enable compression in "user" compression mode */
FI_COMPRESS_RELEASED, /* compressed blocks were released */
FI_ALIGNED_WRITE, /* enable aligned write */
+ FI_QUOTA_INIT_FAIL, /* inidicate failed to initialize quota in drop_inode()/evict_inode() */
FI_MAX, /* max flag, never be used */
};
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
old mode 100644
new mode 100755
index 13deae0..2fb53f54
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -789,9 +789,11 @@ int f2fs_truncate(struct inode *inode)
return -EIO;
}
- err = dquot_initialize(inode);
- if (err)
- return err;
+ if (!is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL)) {
+ err = dquot_initialize(inode);
+ if (err)
+ return err;
+ }
/* we should check inline_data size */
if (!f2fs_may_inline_data(inode)) {
diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
old mode 100644
new mode 100755
index 1213f15..16cf50c
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -758,6 +758,7 @@ void f2fs_evict_inode(struct inode *inode)
if (err) {
err = 0;
set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+ set_inode_flag(inode, FI_QUOTA_INIT_FAIL);
}
f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
@@ -770,6 +771,8 @@ void f2fs_evict_inode(struct inode *inode)
retry:
if (F2FS_HAS_BLOCKS(inode))
err = f2fs_truncate(inode);
+ if (is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL))
+ clear_inode_flag(inode, FI_QUOTA_INIT_FAIL);
if (time_to_inject(sbi, FAULT_EVICT_INODE)) {
f2fs_show_injection_info(sbi, FAULT_EVICT_INODE);
Thanks,
> In fact, I picked up the idea from here.
>
> err = dquot_initialize(inode);
> if (err) {
> err = 0;
> set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
> }
>
> f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
> f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
> f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
>
> sb_start_intwrite(inode->i_sb);
> set_inode_flag(inode, FI_NO_ALLOC);
> i_size_write(inode, 0);
> retry:
> if (F2FS_HAS_BLOCKS(inode))
> err = f2fs_truncate(inode);
>
> Yes, I doubt the problem may also happen in f2fs_evict_inode() with below
> callpath:
>
> - evict_inode
> - dquot_initialize failed
> - sb_start_intwrite
> - f2fs_truncate
> - dquot_initialize lock dqio_sem
>
> How about this?
>
Got it~
Then we need this in both f2fs_evict_inode() and f2fs_drop_inode().
Thanks,
> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
> old mode 100644
> new mode 100755
> index b24b9bc..0e49593
> --- a/fs/f2fs/f2fs.h
> +++ b/fs/f2fs/f2fs.h
> @@ -728,6 +728,7 @@ enum {
> FI_ENABLE_COMPRESS, /* enable compression in "user" compression mode */
> FI_COMPRESS_RELEASED, /* compressed blocks were released */
> FI_ALIGNED_WRITE, /* enable aligned write */
> + FI_QUOTA_INIT_FAIL, /* inidicate failed to initialize quota in drop_inode()/evict_inode() */
> FI_MAX, /* max flag, never be used */
> };
>
> diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
> old mode 100644
> new mode 100755
> index 13deae0..2fb53f54
> --- a/fs/f2fs/file.c
> +++ b/fs/f2fs/file.c
> @@ -789,9 +789,11 @@ int f2fs_truncate(struct inode *inode)
> return -EIO;
> }
>
> - err = dquot_initialize(inode);
> - if (err)
> - return err;
> + if (!is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL)) {
> + err = dquot_initialize(inode);
> + if (err)
> + return err;
> + }
>
> /* we should check inline_data size */
> if (!f2fs_may_inline_data(inode)) {
> diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
> old mode 100644
> new mode 100755
> index 1213f15..16cf50c
> --- a/fs/f2fs/inode.c
> +++ b/fs/f2fs/inode.c
> @@ -758,6 +758,7 @@ void f2fs_evict_inode(struct inode *inode)
> if (err) {
> err = 0;
> set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
> + set_inode_flag(inode, FI_QUOTA_INIT_FAIL);
> }
>
> f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
> @@ -770,6 +771,8 @@ void f2fs_evict_inode(struct inode *inode)
> retry:
> if (F2FS_HAS_BLOCKS(inode))
> err = f2fs_truncate(inode);
> + if (is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL))
> + clear_inode_flag(inode, FI_QUOTA_INIT_FAIL);
>
> if (time_to_inject(sbi, FAULT_EVICT_INODE)) {
> f2fs_show_injection_info(sbi, FAULT_EVICT_INODE);
>
> Thanks,
>
>
> > In fact, I picked up the idea from here.
> >
> > err = dquot_initialize(inode);
> > if (err) {
> > err = 0;
> > set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
> > }
> >
> > f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
> > f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
> > f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
> >
> > sb_start_intwrite(inode->i_sb);
> > set_inode_flag(inode, FI_NO_ALLOC);
> > i_size_write(inode, 0);
> > retry:
> > if (F2FS_HAS_BLOCKS(inode))
> > err = f2fs_truncate(inode);
> >
On Tue, Oct 26, 2021 at 10:56 AM Daeho Jeong <[email protected]> wrote:
>
> > Yes, I doubt the problem may also happen in f2fs_evict_inode() with below
> > callpath:
> >
> > - evict_inode
> > - dquot_initialize failed
> > - sb_start_intwrite
> > - f2fs_truncate
> > - dquot_initialize lock dqio_sem
> >
> > How about this?
> >
>
> Got it~
> Then we need this in both f2fs_evict_inode() and f2fs_drop_inode().
>
It turns out this deadlock issue was related to the Android kernel only. :(
That was related to one of Android tracepoints, which triggered
internal memory reclaim inside of it.
We made a workaround for that in Android kernel.
Thanks,