Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f152b853-6f22-bce5-a1b2-3ea29310f628@bytedance.com>
Date:   Wed, 11 Jan 2023 11:18:13 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.2.2
Subject: Re: [External] Re: [PATCH v4] blk-throtl: Introduce sync and async
 queues for blk-throtl
To:     Tejun Heo <tj@kernel.org>
Cc:     josef@toxicpanda.com, axboe@kernel.dk, cgroups@vger.kernel.org,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        yinxin.x@bytedance.com
References: <20230107130738.75640-1-hanjinke.666@bytedance.com>
 <Y7x7yq5YmcXhVkQf@slm.duckdns.org>
 <b4cf040e-a9d9-8b7a-10cf-80b01d02848f@bytedance.com>
 <Y73k04BhpY/SuwGB@slm.duckdns.org>
From:   hanjinke <hanjinke.666@bytedance.com>
In-Reply-To: <Y73k04BhpY/SuwGB@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk


在 2023/1/11 上午6:21, Tejun Heo 写道:
> On Tue, Jan 10, 2023 at 11:59:09PM +0800, hanjinke wrote:
>>
>>
>> 在 2023/1/10 上午4:40, Tejun Heo 写道:
>>> On Sat, Jan 07, 2023 at 09:07:38PM +0800, Jinke Han wrote:
>>>> + * Assumed that there were only bios queued in ASYNC queue and the SYNC
>>>> + * queue was empty. The ASYNC bio was selected to dispatch and the
>>>> + * disp_sync_cnt was set to 0 after each dispatching. If a ASYNC bio
>>>> + * can't be dispatched because of overlimit in current slice, the process
>>>> + * of dispatch should give up and the spin lock of the request queue
>>>> + * may be released. A new SYNC bio may be queued in the SYNC queue then.
>>>> + * When it's time to dispatch this tg, the SYNC bio was selected and pop
>>>> + * to dispatch as the disp_sync_cnt is 0 and the SYNC queue is no-empty.
>>>> + * If the dispatched bio is smaller than the waiting bio, the bandwidth
>>>> + * may be hard to satisfied as the slice may be trimed after each dispatch.
>>>
>>> I still can't make a good sense of this scenario. Can you give concrete
>>> example scenarios with IOs and why it would matter?
>>>
>>> Thanks.
>>>
>>
>> Assumed that there are many buffer write bios queued in ASYNC queue and the
>> SYNC queue is empty. The buffer write bios are all 1M in size and the bps
>> limit is 1M/s. The throtl_slice is 100ms.
>>
>> Assumed that the start/end_slice is [jiffies1, jiffies1+100] and
>> bytes_disp[w] = 0. The next ASYNC bio can't dispatch because of overlimit
>> within this slice. The wait time is 900ms and the slice will be extended to
>> [jiffies1, jiffies1 + 1000] in tg_may_dispatch.
>>
>> During the waiting of the ASYNC bio, a SYNC 4k bio be queued in SYNC queue.
>> After 900ms, it's time to dispatch the ASYNC io, but the SYNC 4k bio be
>> selected to be dispatched. Now the slice is [jiffies1, jiffies1+1000] and
>> the byte_disp[w] = 4k. The slice may be extended to
>> [jiffies1, jiffies1+1100]. In tg_dispatch_one_bio, the slice will be trimed
>> to [jiffies1+1000, jiffies1+1100], the byte_disp[w] will set 0.
>>
>> After the 4k SYNC bio be dispatched, the WAITING ASYNC bio still cann't
>> be dispatched because of overlimit within this slice.
>>
>> The same thing may happen DISPACH_SYNC_FACTOR times if alway there is a SYNC
>> bio be queued in the SYNC queue when the ASYNC bio is waiting.
>>
>> This means that in nearly 5s, we have dispathed 4 4k SYNC bios and a 1m
>> ASYNC bio.
>>
>> In our test, with 100M/s bps limit setted, the bps only reach to ~80m/s
>> when a fio generate buffer write ios and fsync continuous generated by
>> dbench in same cgroup.
> 
> That makes sense to me. Thanks for the explanation. It'd probably a good
> idea to expand the explanation in the comment tho.
> 
> Thanks.
> 

I will be glad to do so and send the v5.

Thanks.
Jinke