Received-SPF: pass (google.com: domain of linux-kernel+bounces-70131-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Subject: Re: [PATCH v5 01/14] md: don't ignore suspended array in
 md_check_recovery()
To: Xiao Ni <xni@redhat.com>, Yu Kuai <yukuai1@huaweicloud.com>
Cc: mpatocka@redhat.com, heinzm@redhat.com, blazej.kucman@linux.intel.com,
 agk@redhat.com, snitzer@kernel.org, dm-devel@lists.linux.dev,
 song@kernel.org, jbrassow@f14.redhat.com, neilb@suse.de, shli@fb.com,
 akpm@osdl.org, linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org,
 yi.zhang@huawei.com, yangerkun@huawei.com, "yukuai (C)" <yukuai3@huawei.com>
References: <20240201092559.910982-1-yukuai1@huaweicloud.com>
 <20240201092559.910982-2-yukuai1@huaweicloud.com>
 <CALTww2-ZhRBJOD3jXs=xKFaD=iR=dtoC9h2rUQi5Stpi+tJ9Bw@mail.gmail.com>
From: Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <64d27757-9387-09dc-48e8-a9eedd67f075@huaweicloud.com>
Date: Sun, 18 Feb 2024 09:14:59 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CALTww2-ZhRBJOD3jXs=xKFaD=iR=dtoC9h2rUQi5Stpi+tJ9Bw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

在 2024/02/16 14:58, Xiao Ni 写道:
> On Thu, Feb 1, 2024 at 5:30 PM Yu Kuai <yukuai1@huaweicloud.com> wrote:
>>
>> From: Yu Kuai <yukuai3@huawei.com>
>>
>> mddev_suspend() never stop sync_thread, hence it doesn't make sense to
>> ignore suspended array in md_check_recovery(), which might cause
>> sync_thread can't be unregistered.
>>
>> After commit f52f5c71f3d4 ("md: fix stopping sync thread"), following
>> hang can be triggered by test shell/integrity-caching.sh:
> 
> Hi Kuai
> 
> After applying this patch, it's still stuck at mddev_suspend. Maybe
> the deadlock can be fixed by other patches from the patch set. But
> this patch can't fix this issue. If so, the comment is not right.

This patch alone can't fix the problem that mddev_suspend() can stuck
thoroughly, patches 1-4 will all be needed.

Thanks,
Kuai

> 
>>
>> 1) suspend the array:
>> raid_postsuspend
>>   mddev_suspend
>>
>> 2) stop the array:
>> raid_dtr
>>   md_stop
>>    __md_stop_writes
>>     stop_sync_thread
>>      set_bit(MD_RECOVERY_INTR, &mddev->recovery);
>>      md_wakeup_thread_directly(mddev->sync_thread);
>>      wait_event(..., !test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
>>
>> 3) sync thread done:
>> md_do_sync
>>   set_bit(MD_RECOVERY_DONE, &mddev->recovery);
>>   md_wakeup_thread(mddev->thread);
>>
>> 4) daemon thread can't unregister sync thread:
>> md_check_recovery
>>   if (mddev->suspended)
>>     return; -> return directly
>>   md_read_sync_thread
>>   clear_bit(MD_RECOVERY_RUNNING, &mddev->recovery);
>>   -> MD_RECOVERY_RUNNING can't be cleared, hence step 2 hang;
> 
> I add some debug logs when stopping dmraid with lvremove command. The
> step you mentioned are sequential but not async. The process is :
> dev_remove->dm_destroy->__dm_destroy->dm_table_postsuspend_targets(raid_postsuspend)
> -> dm_table_destroy(raid_dtr). It looks like mddev_suspend is waiting
> for active_io to be zero.
> 
> Best Regards
> Xiao
> 
>> This problem is not just related to dm-raid, fix it by ignoring
>> suspended array in md_check_recovery(). And follow up patches will
>> improve dm-raid better to frozen sync thread during suspend.
>>
>> Reported-by: Mikulas Patocka <mpatocka@redhat.com>
>> Closes: https://lore.kernel.org/all/8fb335e-6d2c-dbb5-d7-ded8db5145a@redhat.com/
>> Fixes: 68866e425be2 ("MD: no sync IO while suspended")
>> Fixes: f52f5c71f3d4 ("md: fix stopping sync thread")
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> ---
>>   drivers/md/md.c | 3 ---
>>   1 file changed, 3 deletions(-)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 2266358d8074..07b80278eaa5 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -9469,9 +9469,6 @@ static void md_start_sync(struct work_struct *ws)
>>    */
>>   void md_check_recovery(struct mddev *mddev)
>>   {
>> -       if (READ_ONCE(mddev->suspended))
>> -               return;
>> -
>>          if (mddev->bitmap)
>>                  md_bitmap_daemon_work(mddev);
>>
>> --
>> 2.39.2
>>
> 
> 
> .
>