Received-SPF: pass (google.com: domain of linux-kernel+bounces-70240-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Subject: Re: [PATCH v5 03/14] md: make sure md_do_sync() will set
 MD_RECOVERY_DONE
To: Xiao Ni <xni@redhat.com>, Yu Kuai <yukuai1@huaweicloud.com>
Cc: mpatocka@redhat.com, heinzm@redhat.com, blazej.kucman@linux.intel.com,
 agk@redhat.com, snitzer@kernel.org, dm-devel@lists.linux.dev,
 song@kernel.org, jbrassow@f14.redhat.com, neilb@suse.de, shli@fb.com,
 akpm@osdl.org, linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org,
 yi.zhang@huawei.com, yangerkun@huawei.com, "yukuai (C)" <yukuai3@huawei.com>
References: <20240201092559.910982-1-yukuai1@huaweicloud.com>
 <20240201092559.910982-4-yukuai1@huaweicloud.com>
 <CALTww283nysUDy=jmW4w45GbS6O2nS0XLYX=KEiO2BUp5+cLaA@mail.gmail.com>
From: Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <15f0f260-3a2f-6d9b-e60e-c534a9a4d7d0@huaweicloud.com>
Date: Sun, 18 Feb 2024 14:51:29 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CALTww283nysUDy=jmW4w45GbS6O2nS0XLYX=KEiO2BUp5+cLaA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

在 2024/02/18 13:56, Xiao Ni 写道:
> On Thu, Feb 1, 2024 at 5:30 PM Yu Kuai <yukuai1@huaweicloud.com> wrote:
>>
>> From: Yu Kuai <yukuai3@huawei.com>
>>
>> stop_sync_thread() will interrupt md_do_sync(), and md_do_sync() must
>> set MD_RECOVERY_DONE, so that follow up md_check_recovery() will
>> unregister sync_thread, clear MD_RECOVERY_RUNNING and wake up
>> stop_sync_thread().
>>
>> If MD_RECOVERY_WAIT is set or the array is read-only, md_do_sync() will
>> return without setting MD_RECOVERY_DONE, and after commit f52f5c71f3d4
>> ("md: fix stopping sync thread"), dm-raid switch from
>> md_reap_sync_thread() to stop_sync_thread() to unregister sync_thread
>> from md_stop() and md_stop_writes(), causing the test
>> shell/lvconvert-raid-reshape.sh hang.
>>
>> We shouldn't switch back to md_reap_sync_thread() because it's
>> problematic in the first place. Fix the problem by making sure
>> md_do_sync() will set MD_RECOVERY_DONE.
>>
>> Reported-by: Mikulas Patocka <mpatocka@redhat.com>
>> Closes: https://lore.kernel.org/all/ece2b06f-d647-6613-a534-ff4c9bec1142@redhat.com/
>> Fixes: d5d885fd514f ("md: introduce new personality funciton start()")
>> Fixes: 5fd6c1dce06e ("[PATCH] md: allow checkpoint of recovery with version-1 superblock")
>> Fixes: f52f5c71f3d4 ("md: fix stopping sync thread")
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> ---
>>   drivers/md/md.c | 12 ++++++++----
>>   1 file changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 6906d023f1d6..c65dfd156090 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -8788,12 +8788,16 @@ void md_do_sync(struct md_thread *thread)
>>          int ret;
>>
>>          /* just incase thread restarts... */
>> -       if (test_bit(MD_RECOVERY_DONE, &mddev->recovery) ||
>> -           test_bit(MD_RECOVERY_WAIT, &mddev->recovery))
>> +       if (test_bit(MD_RECOVERY_DONE, &mddev->recovery))
>>                  return;
>> -       if (!md_is_rdwr(mddev)) {/* never try to sync a read-only array */
>> +
>> +       if (test_bit(MD_RECOVERY_INTR, &mddev->recovery))
>> +               goto skip;
>> +
>> +       if (test_bit(MD_RECOVERY_WAIT, &mddev->recovery) ||
>> +           !md_is_rdwr(mddev)) {/* never try to sync a read-only array */
>>                  set_bit(MD_RECOVERY_INTR, &mddev->recovery);
>> -               return;
>> +               goto skip;
>>          }
> 
> Hi all
> 
> I have a question here. The codes above means if MD_RECOVERY_WAIT is
> set, it sets MD_RECOVERY_INTR. If so, the sync thread can't happen.
> But from the codes in md_start function:
> 
>                  set_bit(MD_RECOVERY_WAIT, &mddev->recovery);
>                  md_wakeup_thread(mddev->thread);
>                  ret = mddev->pers->start(mddev);
>                  clear_bit(MD_RECOVERY_WAIT, &mddev->recovery);
>                  md_wakeup_thread(mddev->sync_thread);
> 
> MD_RECOVERY_WAIT means "it'll run sync thread later not interrupt it".
> I guess this patch can introduce a new bug for raid5 journal?

I'm not sure what kind of problem you're talking about. After patch 4,
md_start_sync() should be the only place to register sync_thread, hence
md_start() should not see registered sync_thread. Perhaps
MD_RECOVERY_WAIT and md_wakeup_thread(mddev->sync_thread) can be removed
after patch 4?

> 
> And to resolve this deadlock, we can use this patch:
> 
> --- a/drivers/md/dm-raid.c
> +++ b/drivers/md/dm-raid.c
> @@ -3796,8 +3796,10 @@ static void raid_postsuspend(struct dm_target *ti)
>          struct raid_set *rs = ti->private;
> 
>          if (!test_and_set_bit(RT_FLAG_RS_SUSPENDED, &rs->runtime_flags)) {
> +               if (test_bit(MD_RECOVERY_WAIT, &rs->md.recovery))
> +                       clear_bit(MD_RECOVERY_WAIT, &rs->md.recovery);

You must make sure md_do_sync() is called after this if sync_thread is 
already registered, and I don't understand yet how this is guranteed. :(

Thanks,
Kuai

> 
> Regards
> Xiao
>>
>>          if (mddev_is_clustered(mddev)) {
>> --
>> 2.39.2
>>
> 
> .
>