Date: Wed, 5 Oct 2016 12:08:33 -0700
From: Shaohua Li <shli@fb.com>
To: Tejun Heo <tj@kernel.org>
CC: Paolo Valente <paolo.valente@unimore.it>,
        Vivek Goyal <vgoyal@redhat.com>, <linux-block@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, Jens Axboe <axboe@fb.com>,
        <Kernel-team@fb.com>, <jmoyer@redhat.com>,
        Mark Brown <broonie@kernel.org>,
        Linus Walleij <linus.walleij@linaro.org>,
        Ulf Hansson <ulf.hansson@linaro.org>
Subject: Re: [PATCH V3 00/11] block-throttle: add .high limit
Message-ID: <20161005190832.GA99272@anikkar-mbp.local.dhcp.thefacebook.com>
References: <20161004172852.GB73678@anikkar-mbp.local.dhcp.thefacebook.com>
 <EE90477C-FD56-48C1-8A45-97AF39A8BB0C@unimore.it>
 <20161004185413.GF4205@htj.duckdns.org>
 <F9A2F063-F5A2-49A6-8DFE-91ACDE380645@unimore.it>
 <20161004191427.GG4205@htj.duckdns.org>
 <B8B66027-D652-46E2-A0C6-81F227817571@unimore.it>
 <20161004202754.GJ4205@htj.duckdns.org>
 <257945FA-6789-4D80-8DA3-AC75640C71AE@unimore.it>
 <20161005144946.GA26977@htj.duckdns.org>
 <20161005183052.GA97491@anikkar-mbp.local.dhcp.thefacebook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20161005183052.GA97491@anikkar-mbp.local.dhcp.thefacebook.com>
User-Agent: Mutt/1.6.1 (2016-04-27)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Oct 2016 19:08:44.0037 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR15MB0408
X-OriginatorOrg: fb.com
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-05_05:,,
 signatures=0
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3213
Lines: 96

On Wed, Oct 05, 2016 at 11:30:53AM -0700, Shaohua Li wrote:
> On Wed, Oct 05, 2016 at 10:49:46AM -0400, Tejun Heo wrote:
> > Hello, Paolo.
> > 
> > On Wed, Oct 05, 2016 at 02:37:00PM +0200, Paolo Valente wrote:
> > > In this respect, for your generic, unpredictable scenario to make
> > > sense, there must exist at least one real system that meets the
> > > requirements of such a scenario.  Or, if such a real system does not
> > > yet exist, it must be possible to emulate it.  If it is impossible to
> > > achieve this last goal either, then I miss the usefulness
> > > of looking for solutions for such a scenario.
> > > 
> > > That said, let's define the instance(s) of the scenario that you find
> > > most representative, and let's test BFQ on it/them.  Numbers will give
> > > us the answers.  For example, what about all or part of the following
> > > groups:
> > > . one cyclically doing random I/O for some second and then sequential I/O
> > > for the next seconds
> > > . one doing, say, quasi-sequential I/O in ON/OFF cycles
> > > . one starting an application cyclically
> > > . one playing back or streaming a movie
> > > 
> > > For each group, we could then measure the time needed to complete each
> > > phase of I/O in each cycle, plus the responsiveness in the group
> > > starting an application, plus the frame drop in the group streaming
> > > the movie.  In addition, we can measure the bandwidth/iops enjoyed by
> > > each group, plus, of course, the aggregate throughput of the whole
> > > system.  In particular we could compare results with throttling, BFQ,
> > > and CFQ.
> > > 
> > > Then we could write resulting numbers on the stone, and stick to them
> > > until something proves them wrong.
> > > 
> > > What do you (or others) think about it?
> > 
> > That sounds great and yeah it's lame that we didn't start with that.
> > Shaohua, would it be difficult to compare how bfq performs against
> > blk-throttle?
> 
> I had a test of BFQ. I'm using BFQ found at
> http://algogroup.unimore.it/people/paolo/disk_sched/sources.php. version is
> 4.7.0-v8r3. It's a LSI SSD, queue depth 32. I use default setting. fio script
> is:
> 
> [global]
> ioengine=libaio
> direct=1
> readwrite=randread
> bs=4k
> runtime=60
> time_based=1
> file_service_type=random:36
> overwrite=1
> thread=0
> group_reporting=1
> filename=/dev/sdb
> iodepth=1
> numjobs=8
> 
> [groupA]
> prio=2
> 
> [groupB]
> new_group
> prio=6
> 
> I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.
> 
> iodepth=1 numjobs=1 prio 4:4
> CFQ: 28:28 BFQ: 21:21 deadline: 29:29
> 
> iodepth=8 numjobs=1 prio 4:4
> CFQ: 162:162 BFQ: 102:98 deadline: 205:205
> 
> iodepth=1 numjobs=8 prio 4:4
> CFQ: 157:157 BFQ: 81:92 deadline: 196:197
> 
> iodepth=1 numjobs=1 prio 2:6
> CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29
> 
> iodepth=8 numjobs=1 prio 2:6
> CFQ: 166:174 BFQ: 139:72  deadline: 202:202
> 
> iodepth=1 numjobs=8 prio 2:6
> CFQ: 148:150 BFQ: 90:77 deadline: 198:197

More tests:

iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit
CFQ:51:207  BFQ: 51:45  deadline: 51:216

iodepth=1 numjobs=1 prio 2:6, group A bs=4k, group B bs=64k
CFQ:25:249  BFQ: 23:42  deadline: 26:251

Thanks,
Shaohua