Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH -next v2 2/2] block, bfq: make bfq_has_work() more
 accurate
From:   Paolo Valente <paolo.valente@linaro.org>
In-Reply-To: <740D270D-8723-4399-82CC-26CD861843D7@linaro.org>
Date:   Tue, 17 May 2022 17:06:22 +0200
Cc:     Yu Kuai <yukuai3@huawei.com>, Jens Axboe <axboe@kernel.dk>,
        linux-block <linux-block@vger.kernel.org>,
        linux-kernel@vger.kernel.org, yi.zhang@huawei.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <22FEB802-2872-45A7-8ED8-2DE7D0D5E6CD@linaro.org>
References: <20220513023507.2625717-1-yukuai3@huawei.com>
 <20220513023507.2625717-3-yukuai3@huawei.com>
 <20220516095620.ge5gxmwrnbanfqea@quack3.lan>
 <740D270D-8723-4399-82CC-26CD861843D7@linaro.org>
To:     Jan Kara <jack@suse.cz>
Precedence: bulk


> Il giorno 17 mag 2022, alle ore 16:21, Paolo Valente =
<paolo.valente@linaro.org> ha scritto:
>=20
>=20
>=20
>> Il giorno 16 mag 2022, alle ore 11:56, Jan Kara <jack@suse.cz> ha =
scritto:
>>=20
>> On Fri 13-05-22 10:35:07, Yu Kuai wrote:
>>> bfq_has_work() is using busy_queues currently, which is not accurate
>>> because bfq_queue is busy doesn't represent that it has requests. =
Since
>>> bfqd aready has a counter 'queued' to record how many requests are =
in
>>> bfq, use it instead of busy_queues.
>>>=20
>=20
> The number of requests queued is not equal to the number of busy
> queues (it is >=3D).

No, sorry. It is actually !=3D in general.

In particular, if queued =3D=3D 0 but there are busy queues (although
still waiting for I/O to arrive), then responding that there is no
work caused blk-mq to stop asking, and hence an I/O freeze.  IOW I/O
eventually arrives for a busy queue, but blk-mq does not ask for a new
request any longer.  But maybe things have changed around bfq since
then.

Paolo

>  If this patch is based on this assumption then
> unfortunately it is wrong :(
>=20
> Paolo
>=20
>>> Noted that bfq_has_work() can be called with 'bfqd->lock' held, thus =
the
>>> lock can't be held in bfq_has_work() to protect 'bfqd->queued'.
>>>=20
>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>>=20
>> Looks good. Feel free to add:
>>=20
>> Reviewed-by: Jan Kara <jack@suse.cz>
>>=20
>> 								Honza
>>=20
>>> ---
>>> block/bfq-iosched.c | 16 ++++++++++++----
>>> 1 file changed, 12 insertions(+), 4 deletions(-)
>>>=20
>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>> index 61750696e87f..740dd83853a6 100644
>>> --- a/block/bfq-iosched.c
>>> +++ b/block/bfq-iosched.c
>>> @@ -2210,7 +2210,11 @@ static void bfq_add_request(struct request =
*rq)
>>>=20
>>> 	bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq));
>>> 	bfqq->queued[rq_is_sync(rq)]++;
>>> -	bfqd->queued++;
>>> +	/*
>>> +	 * Updating of 'bfqd->queued' is protected by 'bfqd->lock', =
however, it
>>> +	 * may be read without holding the lock in bfq_has_work().
>>> +	 */
>>> +	WRITE_ONCE(bfqd->queued, bfqd->queued + 1);
>>>=20
>>> 	if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_bfqq_sync(bfqq)) {
>>> 		bfq_check_waker(bfqd, bfqq, now_ns);
>>> @@ -2402,7 +2406,11 @@ static void bfq_remove_request(struct =
request_queue *q,
>>> 	if (rq->queuelist.prev !=3D &rq->queuelist)
>>> 		list_del_init(&rq->queuelist);
>>> 	bfqq->queued[sync]--;
>>> -	bfqd->queued--;
>>> +	/*
>>> +	 * Updating of 'bfqd->queued' is protected by 'bfqd->lock', =
however, it
>>> +	 * may be read without holding the lock in bfq_has_work().
>>> +	 */
>>> +	WRITE_ONCE(bfqd->queued, bfqd->queued - 1);
>>> 	elv_rb_del(&bfqq->sort_list, rq);
>>>=20
>>> 	elv_rqhash_del(q, rq);
>>> @@ -5063,11 +5071,11 @@ static bool bfq_has_work(struct =
blk_mq_hw_ctx *hctx)
>>> 	struct bfq_data *bfqd =3D hctx->queue->elevator->elevator_data;
>>>=20
>>> 	/*
>>> -	 * Avoiding lock: a race on bfqd->busy_queues should cause at
>>> +	 * Avoiding lock: a race on bfqd->queued should cause at
>>> 	 * most a call to dispatch for nothing
>>> 	 */
>>> 	return !list_empty_careful(&bfqd->dispatch) ||
>>> -		bfq_tot_busy_queues(bfqd) > 0;
>>> +		READ_ONCE(bfqd->queued);
>>> }
>>>=20
>>> static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx =
*hctx)
>>> --=20
>>> 2.31.1
>>>=20
>> --=20
>> Jan Kara <jack@suse.com>
>> SUSE Labs, CR
>=20