Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f9b8e682-92aa-c39c-4d91-d77d104e0767@bytedance.com>
Date:   Tue, 10 Jan 2023 21:07:25 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.2.2
Subject: Re: [External] Re: [PATCH v3] blk-throtl: Introduce sync and async
 queues for blk-throtl
To:     Tejun Heo <tj@kernel.org>
Cc:     Jan Kara <jack@suse.cz>,
        =?UTF-8?Q?Michal_Koutn=c3=bd?= <mkoutny@suse.com>,
        josef@toxicpanda.com, axboe@kernel.dk, cgroups@vger.kernel.org,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        yinxin.x@bytedance.com
References: <20221226130505.7186-1-hanjinke.666@bytedance.com>
 <20230105161854.GA1259@blackbody.suse.cz>
 <20230106153813.4ttyuikzaagkk2sc@quack3> <Y7hTHZQYsCX6EHIN@slm.duckdns.org>
 <c839ba6c-80ac-6d92-af64-5c0e1956ae93@bytedance.com>
 <Y7hlX4T1UOmQHiGf@slm.duckdns.org>
 <e499f088-8ed9-2e19-b2e5-efaa4f9738f0@bytedance.com>
 <Y7xYJfRLSMYk9tj9@slm.duckdns.org>
From:   hanjinke <hanjinke.666@bytedance.com>
In-Reply-To: <Y7xYJfRLSMYk9tj9@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk


在 2023/1/10 上午2:08, Tejun Heo 写道:
> Hello,
> 
> On Sat, Jan 07, 2023 at 12:44:35PM +0800, hanjinke wrote:
>> For cost.model setting, We first use the tools iocost provided to test the
>> benchmark model parameters of different types of disks online, and then save
>> these benchmark parameters to a parametric Model Table. During the
>> deployment process, pull and set the corresponding model parameters
>> according to the type of disk.
>>
>> The setting of cost.qos should be considered slightly more，we need to make
>> some compromises between overall disk throughput and io latency.
>> The average disk utilization of the entire disk on a specific business and
>> the RLA（if it is io sensitive） of key businesses will be taken as
>> important input considerations. The cost.qos will be dynamically fine-tuned
>> according to the health status monitoring of key businesses.
> 
> Ah, I see. Do you use the latency targets and min/max ranges or just fixate
> the vrate by setting min == max?

Currently we use the former.

> 
>> For cost.weight setting, high-priority services  will gain greater
>> advantages through weight settings to deal with a large number of io
>> requests in a short period of time. It works fine as work-conservation
>> of iocost works well according to our observation.
> 
> Glad to hear.
> 
>> These practices can be done better and I look forward to your better
>> suggestions.
> 
> It's still in progress but resctl-bench's iocost-tune benchmark is what
> we're starting to use:
> 
>   https://github.com/facebookexperimental/resctl-demo/blob/main/resctl-bench/doc/iocost-tune.md
> 
> The benchmark takes like 6 hours and what it does is probing the whole vrate
> range looking for behavior inflection points given the scenario of
> protecting a latency sensitive workload against memory leak. On completion,
> it provides several solutions based on the behavior observed.
> 
> The benchmark is destructive (to the content on the target ssd) and can be
> tricky to set up. There's installable image to help setting up and running
> the benchmark:
> 
>   https://github.com/iocost-benchmark/resctl-demo-image-recipe/actions
> 
> The eventual goal is collecting these benchmark results in the following git
> repo:
> 
>   https://github.com/iocost-benchmark/iocost-benchmarks
> 
> which generates hwdb files describing all the found solution and make
> systemd apply the appropriate configuration on boot automatically.
> 
> It's still all a work in progress but hopefully we should be able to
> configure iocost reasonably on boot on most SSDs.
> 
> Thanks.
> 

These methodologies are worthy of our study and will definitely help our 
future deployment of iocost. Thanks a lot.

Thanks.