Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Subject: Re: [PATCH -next v2 02/28] md: use 'mddev->suspended' for
 is_md_suspended()
To:     Xiao Ni <xni@redhat.com>, Yu Kuai <yukuai1@huaweicloud.com>
Cc:     agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com,
        song@kernel.org, linux-kernel@vger.kernel.org,
        linux-raid@vger.kernel.org, yi.zhang@huawei.com,
        yangerkun@huawei.com, "yukuai (C)" <yukuai3@huawei.com>
References: <20230828020021.2489641-1-yukuai1@huaweicloud.com>
 <20230828020021.2489641-3-yukuai1@huaweicloud.com>
 <CALTww29iHX_GHogGFEfzdaDjohBr8ycfiz2=E_ru4JYvmrTYdA@mail.gmail.com>
From:   Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <1407f90d-d608-6059-7381-adb230585e3d@huaweicloud.com>
Date:   Mon, 25 Sep 2023 09:34:48 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <CALTww29iHX_GHogGFEfzdaDjohBr8ycfiz2=E_ru4JYvmrTYdA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi,

在 2023/09/20 16:46, Xiao Ni 写道:
> On Mon, Aug 28, 2023 at 10:04 AM Yu Kuai <yukuai1@huaweicloud.com> wrote:
>>
>> From: Yu Kuai <yukuai3@huawei.com>
>>
>> 'pers->prepare_suspend' is introduced to prevent a deadlock for raid456,
>> this change prepares to clean this up in later patches while refactoring
>> mddev_suspend(). Specifically allow reshape to make progress while
>> waiting for 'active_io' to be 0.
> 
> Hi Kuai
> 
>>From my side, I can't understand the comments. The change has
> relationship with pers->prepare_suspend? And why this change can
> affect reshape? If this change indeed can affect these two things, can
> you explain more?

First of all, 'prepare_suspend' is used to fix a deadlock in raid456:

1) suspend is waiting for normal io to be done.

mddev_suspend
  mddev->suspended++ -> new sync_thread can't start
  percpu_ref_kill(active_io)
  wait_event(percpu_ref_is_zero(active_io))

2) normal io is waiting for reshape to make progress.
3) reshape is waiting for suspended array to be resumed.

md_check_recovery
  if (is_md_suspended(mddev))
   return

Then prepare_suspend will failed the io that is waiting for reshape to
make progress:

mddev_suspend
  mddev->suspended++
  percpu_ref_kill(active_io)
   -> new io will be stuck in md_handle_request
  pers->prepare_suspend() -> raid5_prepare_suspend
   -> wake_up(wait_for_overlap)
		// woke up
		raid5_make_request
		 make_stripe_request
		  !reshape_inprogress(mddev) && reshape_disabled(mddev)
		   // return io error for the io that is waiting for
		   // reshape to make progress

  wait_event(percpu_ref_is_zero(active_io))

With this patch and the new api to suspend array:

mddev_suspend
  percpu_ref_kill(active_io)
  wait_event(percpu_ref_is_zero(active_io))
  -> while waiting for normal io to be done, new sync_thread can still
     start, and reshape can still make progress.
  mddev->suspended++

> 
>>
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> ---
>>   drivers/md/md.c | 2 +-
>>   drivers/md/md.h | 2 +-
>>   2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 9d8dff9d923c..7fa311a14317 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -355,7 +355,7 @@ static DEFINE_SPINLOCK(all_mddevs_lock);
>>    */
>>   static bool is_suspended(struct mddev *mddev, struct bio *bio)
>>   {
>> -       if (is_md_suspended(mddev))
>> +       if (is_md_suspended(mddev) || percpu_ref_is_dying(&mddev->active_io))
> 
> If we use mddev->suspended to judge if the raid is suspended, it
> should be enough? Because mddev->suspended must be true when active_io
> is dying.

In the new api, active_io is killed before increasing suspended, and the
difference is that the timing that array is suspended will be delayed
from the start of mddev_suspend() to when all dispatched io is done.

I think this is OK because this doesn't change behaviour when
mddev_suspend() returns.

Thanks,
Kuai
`
> 
> Best Regards
> Xiao
>>                  return true;
>>          if (bio_data_dir(bio) != WRITE)
>>                  return false;
>> diff --git a/drivers/md/md.h b/drivers/md/md.h
>> index b628c292506e..fb3b123f16dd 100644
>> --- a/drivers/md/md.h
>> +++ b/drivers/md/md.h
>> @@ -584,7 +584,7 @@ static inline bool md_is_rdwr(struct mddev *mddev)
>>
>>   static inline bool is_md_suspended(struct mddev *mddev)
>>   {
>> -       return percpu_ref_is_dying(&mddev->active_io);
>> +       return READ_ONCE(mddev->suspended);
>>   }
>>
>>   static inline int __must_check mddev_lock(struct mddev *mddev)
>> --
>> 2.39.2
>>
> 
> .
>