Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Sender: Tejun Heo <htejun@gmail.com>
Date:   Mon, 9 Jan 2023 08:08:37 -1000
From:   Tejun Heo <tj@kernel.org>
To:     hanjinke <hanjinke.666@bytedance.com>
Cc:     Jan Kara <jack@suse.cz>,
        Michal =?iso-8859-1?Q?Koutn=FD?= <mkoutny@suse.com>,
        josef@toxicpanda.com, axboe@kernel.dk, cgroups@vger.kernel.org,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        yinxin.x@bytedance.com
Subject: Re: [External] Re: [PATCH v3] blk-throtl: Introduce sync and async
 queues for blk-throtl
Message-ID: <Y7xYJfRLSMYk9tj9@slm.duckdns.org>
References: <20221226130505.7186-1-hanjinke.666@bytedance.com>
 <20230105161854.GA1259@blackbody.suse.cz>
 <20230106153813.4ttyuikzaagkk2sc@quack3>
 <Y7hTHZQYsCX6EHIN@slm.duckdns.org>
 <c839ba6c-80ac-6d92-af64-5c0e1956ae93@bytedance.com>
 <Y7hlX4T1UOmQHiGf@slm.duckdns.org>
 <e499f088-8ed9-2e19-b2e5-efaa4f9738f0@bytedance.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <e499f088-8ed9-2e19-b2e5-efaa4f9738f0@bytedance.com>
Precedence: bulk

Hello,

On Sat, Jan 07, 2023 at 12:44:35PM +0800, hanjinke wrote:
> For cost.model setting, We first use the tools iocost provided to test the
> benchmark model parameters of different types of disks online, and then save
> these benchmark parameters to a parametric Model Table. During the
> deployment process, pull and set the corresponding model parameters
> according to the type of disk.
> 
> The setting of cost.qos should be considered slightly more，we need to make
> some compromises between overall disk throughput and io latency.
> The average disk utilization of the entire disk on a specific business and
> the RLA（if it is io sensitive） of key businesses will be taken as
> important input considerations. The cost.qos will be dynamically fine-tuned
> according to the health status monitoring of key businesses.

Ah, I see. Do you use the latency targets and min/max ranges or just fixate
the vrate by setting min == max?

> For cost.weight setting, high-priority services  will gain greater
> advantages through weight settings to deal with a large number of io
> requests in a short period of time. It works fine as work-conservation
> of iocost works well according to our observation.

Glad to hear.

> These practices can be done better and I look forward to your better
> suggestions.

It's still in progress but resctl-bench's iocost-tune benchmark is what
we're starting to use:

 https://github.com/facebookexperimental/resctl-demo/blob/main/resctl-bench/doc/iocost-tune.md

The benchmark takes like 6 hours and what it does is probing the whole vrate
range looking for behavior inflection points given the scenario of
protecting a latency sensitive workload against memory leak. On completion,
it provides several solutions based on the behavior observed.

The benchmark is destructive (to the content on the target ssd) and can be
tricky to set up. There's installable image to help setting up and running
the benchmark:

 https://github.com/iocost-benchmark/resctl-demo-image-recipe/actions

The eventual goal is collecting these benchmark results in the following git
repo:

 https://github.com/iocost-benchmark/iocost-benchmarks

which generates hwdb files describing all the found solution and make
systemd apply the appropriate configuration on boot automatically.

It's still all a work in progress but hopefully we should be able to
configure iocost reasonably on boot on most SSDs.

Thanks.

-- 
tejun