2020-09-14 10:47:33

by yebin (H)

[permalink] [raw]
Subject: [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks

As we test disk offline/online with running fsstress, we find fsstress
process is keeping running state.
kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
....
kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114

ext4_mb_new_blocks
repeat:
ext4_mb_discard_preallocations_should_retry(sb, ac, &seq)
freed = ext4_mb_discard_preallocations
ext4_mb_discard_group_preallocations
this_cpu_inc(discard_pa_seq);
---> freed == 0
seq_retry = ext4_get_discard_pa_seq_sum
for_each_possible_cpu(__cpu)
__seq += per_cpu(discard_pa_seq, __cpu);
if (seq_retry != *seq) {
*seq = seq_retry;
ret = true;
}

As we see seq_retry is sum of discard_pa_seq every cpu, if
ext4_mb_discard_group_preallocations return zero discard_pa_seq in this
cpu maybe increase one, so condition "seq_retry != *seq" have always
been met.
Ritesh Harjani suggest to in ext4_mb_discard_group_preallocations function we
only increase discard_pa_seq when there is some PA to free.

Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
Signed-off-by: Ye Bin <[email protected]>
---
fs/ext4/mballoc.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 132c118d12e1..ff47347012f4 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4189,7 +4189,6 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
INIT_LIST_HEAD(&list);
repeat:
ext4_lock_group(sb, group);
- this_cpu_inc(discard_pa_seq);
list_for_each_entry_safe(pa, tmp,
&grp->bb_prealloc_list, pa_group_list) {
spin_lock(&pa->pa_lock);
@@ -4206,6 +4205,9 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
/* seems this one can be freed ... */
ext4_mb_mark_pa_deleted(sb, pa);

+ if (!free)
+ this_cpu_inc(discard_pa_seq);
+
/* we can trust pa_free ... */
free += pa->pa_free;

--
2.25.4


2020-09-15 12:14:25

by Jan Kara

[permalink] [raw]
Subject: Re: [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks

On Mon 14-09-20 18:47:42, Ye Bin wrote:
> As we test disk offline/online with running fsstress, we find fsstress
> process is keeping running state.
> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
> ....
> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
>
> ext4_mb_new_blocks
> repeat:
> ext4_mb_discard_preallocations_should_retry(sb, ac, &seq)
> freed = ext4_mb_discard_preallocations
> ext4_mb_discard_group_preallocations
> this_cpu_inc(discard_pa_seq);
> ---> freed == 0
> seq_retry = ext4_get_discard_pa_seq_sum
> for_each_possible_cpu(__cpu)
> __seq += per_cpu(discard_pa_seq, __cpu);
> if (seq_retry != *seq) {
> *seq = seq_retry;
> ret = true;
> }
>
> As we see seq_retry is sum of discard_pa_seq every cpu, if
> ext4_mb_discard_group_preallocations return zero discard_pa_seq in this
> cpu maybe increase one, so condition "seq_retry != *seq" have always
> been met.
> Ritesh Harjani suggest to in ext4_mb_discard_group_preallocations function we
> only increase discard_pa_seq when there is some PA to free.
>
> Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
> Signed-off-by: Ye Bin <[email protected]>

The patch looks good to me. You can add:

Reviewed-by: Jan Kara <[email protected]>

But as I mentioned in my previous reply I also think the attached patch
also needs to be merged to avoid premature ENOSPC errors (which your change
makes somewhat more likely). Ritesh do you agree?

Honza


> ---
> fs/ext4/mballoc.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
> index 132c118d12e1..ff47347012f4 100644
> --- a/fs/ext4/mballoc.c
> +++ b/fs/ext4/mballoc.c
> @@ -4189,7 +4189,6 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
> INIT_LIST_HEAD(&list);
> repeat:
> ext4_lock_group(sb, group);
> - this_cpu_inc(discard_pa_seq);
> list_for_each_entry_safe(pa, tmp,
> &grp->bb_prealloc_list, pa_group_list) {
> spin_lock(&pa->pa_lock);
> @@ -4206,6 +4205,9 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
> /* seems this one can be freed ... */
> ext4_mb_mark_pa_deleted(sb, pa);
>
> + if (!free)
> + this_cpu_inc(discard_pa_seq);
> +
> /* we can trust pa_free ... */
> free += pa->pa_free;
>
> --
> 2.25.4
>
--
Jan Kara <[email protected]>
SUSE Labs, CR


Attachments:
(No filename) (2.72 kB)
0001-ext4-Discard-preallocations-before-releasing-group-l.patch (2.15 kB)
Download all attachments

2020-09-16 20:48:51

by Ritesh Harjani

[permalink] [raw]
Subject: Re: [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks



On 9/15/20 5:41 PM, Jan Kara wrote:
> On Mon 14-09-20 18:47:42, Ye Bin wrote:
>> As we test disk offline/online with running fsstress, we find fsstress
>> process is keeping running state.
>> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
>> ....
>> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
>>
>> ext4_mb_new_blocks
>> repeat:
>> ext4_mb_discard_preallocations_should_retry(sb, ac, &seq)
>> freed = ext4_mb_discard_preallocations
>> ext4_mb_discard_group_preallocations
>> this_cpu_inc(discard_pa_seq);
>> ---> freed == 0
>> seq_retry = ext4_get_discard_pa_seq_sum
>> for_each_possible_cpu(__cpu)
>> __seq += per_cpu(discard_pa_seq, __cpu);
>> if (seq_retry != *seq) {
>> *seq = seq_retry;
>> ret = true;
>> }
>>
>> As we see seq_retry is sum of discard_pa_seq every cpu, if
>> ext4_mb_discard_group_preallocations return zero discard_pa_seq in this
>> cpu maybe increase one, so condition "seq_retry != *seq" have always
>> been met.
>> Ritesh Harjani suggest to in ext4_mb_discard_group_preallocations function we
>> only increase discard_pa_seq when there is some PA to free.

@yebin,
Did you confirm by running your test case that this patch indeed fixes
your reported issue.
With that confirmed, the patch does looks good to me. Feel free to add.

Reviewed-by: Ritesh Harjani <[email protected]>

>>
>> Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
>> Signed-off-by: Ye Bin <[email protected]>
>
> The patch looks good to me. You can add:
>
> Reviewed-by: Jan Kara <[email protected]>
>
> But as I mentioned in my previous reply I also think the attached patch
> also needs to be merged to avoid premature ENOSPC errors (which your change
> makes somewhat more likely). Ritesh do you agree?


Yes, agree that Jan's attached patch should help to avoid premature
ENOSPC errors. We should have his patch too on top of current patch.

@yebin
Should we have a v5 of then, with both patches included for merging?