Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH 3/3] md/raid10: fix io loss while replacement replace rdev
To:     linan666@huaweicloud.com, song@kernel.org, shli@fb.com,
        allenpeng@synology.com, alexwu@synology.com,
        bingjingc@synology.com, neilb@suse.de
Cc:     linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org,
        linan122@huawei.com, yi.zhang@huawei.com, houtao1@huawei.com,
        yangerkun@huawei.com, "yukuai (C)" <yukuai3@huawei.com>
References: <20230522115449.2203939-1-linan666@huaweicloud.com>
 <20230522115449.2203939-4-linan666@huaweicloud.com>
From:   Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <28234b54-f7b1-0cd7-8955-a8ec64bc0212@huaweicloud.com>
Date:   Mon, 22 May 2023 21:03:10 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <20230522115449.2203939-4-linan666@huaweicloud.com>
Content-Type: text/plain; charset=gbk; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi,

?? 2023/05/22 19:54, linan666@huaweicloud.com д??:
> From: Li Nan <linan122@huawei.com>
> 
> When we remove a disk which has replacement, first set rdev to NULL
> and then set replacement to rdev, finally set replacement to NULL (see
> raid10_remove_disk()). If io is submitted during the same time, it might
> read both rdev and replacement as NULL, and io will not be submitted.
> 
>    rdev -> NULL
> 			read rdev
>    replacement -> NULL
> 			read replacement
> 
> Fix it by reading replacement first and rdev later, meanwhile, use smp_mb()
> to prevent memory reordering.

Looks good, feel free to add:

Reviewed-by: Yu Kuai <yukuai3@huawei.com>
> 
> Fixes: 475b0321a4df ("md/raid10: writes should get directed to replacement as well as original.")
> Signed-off-by: Li Nan <linan122@huawei.com>
> ---
>   drivers/md/raid10.c | 22 ++++++++++++++++++----
>   1 file changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 70cc87c7ee57..25a5a7b1e95c 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -779,8 +779,16 @@ static struct md_rdev *read_balance(struct r10conf *conf,
>   		disk = r10_bio->devs[slot].devnum;
>   		rdev = rcu_dereference(conf->mirrors[disk].replacement);
>   		if (rdev == NULL || test_bit(Faulty, &rdev->flags) ||
> -		    r10_bio->devs[slot].addr + sectors > rdev->recovery_offset)
> +		    r10_bio->devs[slot].addr + sectors >
> +		    rdev->recovery_offset) {
> +			/*
> +			 * Read replacement first to prevent reading both rdev
> +			 * and replacement as NULL during replacement replace
> +			 * rdev.
> +			 */
> +			smp_mb();
>   			rdev = rcu_dereference(conf->mirrors[disk].rdev);
> +		    }
>   		if (rdev == NULL ||
>   		    test_bit(Faulty, &rdev->flags))
>   			continue;
> @@ -1479,9 +1487,15 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
>   
>   	for (i = 0;  i < conf->copies; i++) {
>   		int d = r10_bio->devs[i].devnum;
> -		struct md_rdev *rdev = rcu_dereference(conf->mirrors[d].rdev);
> -		struct md_rdev *rrdev = rcu_dereference(
> -			conf->mirrors[d].replacement);
> +		struct md_rdev *rdev, *rrdev;
> +
> +		rrdev = rcu_dereference(conf->mirrors[d].replacement);
> +		/*
> +		 * Read replacement first to Prevent reading both rdev and
> +		 * replacement as NULL during replacement replace rdev.
> +		 */
> +		smp_mb();
> +		rdev = rcu_dereference(conf->mirrors[d].rdev);
>   		if (rdev == rrdev)
>   			rrdev = NULL;
>   		if (rdev && (test_bit(Faulty, &rdev->flags)))
>