Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230621105728.1268542-1-yukuai1@huaweicloud.com>
In-Reply-To: <20230621105728.1268542-1-yukuai1@huaweicloud.com>
From:   Song Liu <song@kernel.org>
Date:   Wed, 21 Jun 2023 16:12:40 -0700
Message-ID: <CAPhsuW5Th2Q9xbP3-ayteNVcHjiPRKpM0agS0u9KP4YG+vxk7g@mail.gmail.com>
Subject: Re: [PATCH v2] raid10: avoid spin_lock from fastpath from raid10_unplug()
To:     Yu Kuai <yukuai1@huaweicloud.com>
Cc:     pmenzel@molgen.mpg.de, linux-raid@vger.kernel.org,
        linux-kernel@vger.kernel.org, yukuai3@huawei.com,
        yi.zhang@huawei.com, yangerkun@huawei.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Jun 20, 2023 at 7:59=E2=80=AFPM Yu Kuai <yukuai1@huaweicloud.com> w=
rote:
>
> From: Yu Kuai <yukuai3@huawei.com>
>
> Commit 0c0be98bbe67 ("md/raid10: prevent unnecessary calls to wake_up()
> in fast path") missed one place, for example, with:
>
>         fio -direct=3D1 -rw=3Dwrite/randwrite -iodepth=3D1 ...
>
> Plug and unplug are called for each io, then wake_up() from raid10_unplug=
()
> will cause lock contention as well.
>
> Avoid this contention by using wake_up_barrier() instead of wake_up(),
> where spin_lock is not held if waitqueue is empty.
>
> Fio test script:
>
> [global]
> name=3Drandom reads and writes
> ioengine=3Dlibaio
> direct=3D1
> readwrite=3Drandrw
> rwmixread=3D70
> iodepth=3D64
> buffered=3D0
> filename=3D/dev/md0
> size=3D1G
> runtime=3D30
> time_based
> randrepeat=3D0
> norandommap
> refill_buffers
> ramp_time=3D10
> bs=3D4k
> numjobs=3D400
> group_reporting=3D1
> [job1]
>
> Test result with ramdisk raid10(By Ali):
>
>         Before this patch       With this patch
> READ    IOPS=3D2033k              IOPS=3D3642k
> WRITE   IOPS=3D871k               IOPS=3D1561K
>
> By the way, in this scenario, blk_plug_cb() will be allocated and freed
> for each io, this seems need to be optimized as well.
>
> Reported-and-tested-by: Ali Gholami Rudi <aligrudi@gmail.com>
> Closes: https://lore.kernel.org/all/20231606122233@laper.mirepesht/
> Signed-off-by: Yu Kuai <yukuai3@huawei.com>

Applied to md-next.

Thanks,
Song

> ---
> Changes in v2:
>  - update commit message;
>
>  drivers/md/raid10.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index d0de8c9fb3cf..fbaaa5e05edc 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -1118,7 +1118,7 @@ static void raid10_unplug(struct blk_plug_cb *cb, b=
ool from_schedule)
>                 spin_lock_irq(&conf->device_lock);
>                 bio_list_merge(&conf->pending_bio_list, &plug->pending);
>                 spin_unlock_irq(&conf->device_lock);
> -               wake_up(&conf->wait_barrier);
> +               wake_up_barrier(conf);
>                 md_wakeup_thread(mddev->thread);
>                 kfree(plug);
>                 return;
> @@ -1127,7 +1127,7 @@ static void raid10_unplug(struct blk_plug_cb *cb, b=
ool from_schedule)
>         /* we aren't scheduling, so we can do the write-out directly. */
>         bio =3D bio_list_get(&plug->pending);
>         raid1_prepare_flush_writes(mddev->bitmap);
> -       wake_up(&conf->wait_barrier);
> +       wake_up_barrier(conf);
>
>         while (bio) { /* submit pending writes */
>                 struct bio *next =3D bio->bi_next;
> --
> 2.39.2
>