Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b4cf040e-a9d9-8b7a-10cf-80b01d02848f@bytedance.com>
Date:   Tue, 10 Jan 2023 23:59:09 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.2.2
Subject: Re: [External] Re: [PATCH v4] blk-throtl: Introduce sync and async
 queues for blk-throtl
To:     Tejun Heo <tj@kernel.org>
Cc:     josef@toxicpanda.com, axboe@kernel.dk, cgroups@vger.kernel.org,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        yinxin.x@bytedance.com
References: <20230107130738.75640-1-hanjinke.666@bytedance.com>
 <Y7x7yq5YmcXhVkQf@slm.duckdns.org>
From:   hanjinke <hanjinke.666@bytedance.com>
In-Reply-To: <Y7x7yq5YmcXhVkQf@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk


在 2023/1/10 上午4:40, Tejun Heo 写道:
> On Sat, Jan 07, 2023 at 09:07:38PM +0800, Jinke Han wrote:
>> + * Assumed that there were only bios queued in ASYNC queue and the SYNC
>> + * queue was empty. The ASYNC bio was selected to dispatch and the
>> + * disp_sync_cnt was set to 0 after each dispatching. If a ASYNC bio
>> + * can't be dispatched because of overlimit in current slice, the process
>> + * of dispatch should give up and the spin lock of the request queue
>> + * may be released. A new SYNC bio may be queued in the SYNC queue then.
>> + * When it's time to dispatch this tg, the SYNC bio was selected and pop
>> + * to dispatch as the disp_sync_cnt is 0 and the SYNC queue is no-empty.
>> + * If the dispatched bio is smaller than the waiting bio, the bandwidth
>> + * may be hard to satisfied as the slice may be trimed after each dispatch.
> 
> I still can't make a good sense of this scenario. Can you give concrete
> example scenarios with IOs and why it would matter?
> 
> Thanks.
> 

Assumed that there are many buffer write bios queued in ASYNC queue and 
the SYNC queue is empty. The buffer write bios are all 1M in size and 
the bps limit is 1M/s. The throtl_slice is 100ms.

Assumed that the start/end_slice is [jiffies1, jiffies1+100] and 
bytes_disp[w] = 0. The next ASYNC bio can't dispatch because of 
overlimit within this slice. The wait time is 900ms and the slice will 
be extended to [jiffies1, jiffies1 + 1000] in tg_may_dispatch.

During the waiting of the ASYNC bio, a SYNC 4k bio be queued in SYNC 
queue. After 900ms, it's time to dispatch the ASYNC io, but the SYNC 4k 
bio be selected to be dispatched. Now the slice is [jiffies1, 
jiffies1+1000] and the byte_disp[w] = 4k. The slice may be extended to
[jiffies1, jiffies1+1100]. In tg_dispatch_one_bio, the slice will be 
trimed to [jiffies1+1000, jiffies1+1100], the byte_disp[w] will set 0.

After the 4k SYNC bio be dispatched, the WAITING ASYNC bio still cann't
be dispatched because of overlimit within this slice.

The same thing may happen DISPACH_SYNC_FACTOR times if alway there is a 
SYNC bio be queued in the SYNC queue when the ASYNC bio is waiting.

This means that in nearly 5s, we have dispathed 4 4k SYNC bios and a 1m 
ASYNC bio.

In our test, with 100M/s bps limit setted, the bps only reach to ~80m/s
when a fio generate buffer write ios and fsync continuous generated by 
dbench in same cgroup.

Thanks
Jinke.