Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753966AbcJDR3X (ORCPT ); Tue, 4 Oct 2016 13:29:23 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:53960 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752537AbcJDR3U (ORCPT ); Tue, 4 Oct 2016 13:29:20 -0400 Date: Tue, 4 Oct 2016 10:28:53 -0700 From: Shaohua Li To: Paolo Valente CC: Tejun Heo , Vivek Goyal , , , Jens Axboe , , , Mark Brown , Linus Walleij , Ulf Hansson Subject: Re: [PATCH V3 00/11] block-throttle: add .high limit Message-ID: <20161004172852.GB73678@anikkar-mbp.local.dhcp.thefacebook.com> References: <20161004132805.GB28808@redhat.com> <20161004155616.GB4205@htj.duckdns.org> <20161004162759.GD4205@htj.duckdns.org> <278BCC7B-ED58-4FDF-9243-FAFC3F862E4D@unimore.it> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <278BCC7B-ED58-4FDF-9243-FAFC3F862E4D@unimore.it> User-Agent: Mutt/1.6.1 (2016-04-27) X-Originating-IP: [2620:10d:c090:180::9298] X-ClientProxiedBy: DM3PR14CA0053.namprd14.prod.outlook.com (10.166.156.149) To BY2PR15MB0407.namprd15.prod.outlook.com (10.163.109.25) X-MS-Office365-Filtering-Correlation-Id: 96689ce1-da7a-4f34-8cba-08d3ec7bf13d X-Microsoft-Exchange-Diagnostics: 1;BY2PR15MB0407;2:onkZVXvuXkfZ7X4cp1drm1tqzqOLH0YgShdeEJgQCmydurQsL9EuytB7UgzgP+AlAewfF1+Qh47ITsO1Vqch5R1zh2cyxBjCjxQ7oW4yLrEoBL8ItaNQatusDDqBFJcf3E8jsJX6hxN7izwMRC2jZtaCw/ROytNx4QHqHG32c7yJAuPQgXcnCnPBHEzviebgkqRTUvMqJicAwDnsCzr4Zg==;3:FaS1o6L+6kJ+erqos1xk7YLGDtfhSuVfKxEc3Abhu3wQoXrbFW8MGifufcSIxXXhteCbFpVYJmoZiehMhfHhrDy+JxeGzOhQ7rQv3QypN3ZkbYQHyzmqk14SCt4jl7twqUNIoeIxWrYVudkeG+bwWQ==;25:f0bXBR6lVg31luS2/2Vlo8Fv1X7GvZfXWhN9QKk3ws4xDINOqNd+58XU9h2Xi49wtxBxZASVYrR5Gg+Xx0hhmFRRmVK+WaVTXJZ3/TK6IWRi8+TJMSaH3S2m9oVl+w08QQr2HCO1PH0Log3hC2SX9kbZZ80WuZCVYnatIhp5lJqHdMwJb5ibEkCpyeEO4Pd2esgkTUapMH0sJrNXnoRDw2cD1ku/wyClDbRx9zUhtuNiBVNWJ8OjVwUlGpFsCCrbhKdfw0+Q/FxYr1rrrdUd7d5WE3/OFTlM9lX3ICSCMSS+eWu1NSdRNvevL06ek1dykQyaaETbkuO4BirzcICWBuxRVWOzgagcXHYxBLhBQaDeJVa0beLAkvQaJTUDpVKWU5Wdauj1eXhS5AZ3AoOEMib/J/eRiXQ7cnjPpgSruy7AkKaNJefzoB2SfTqM+58Q X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR15MB0407; X-Microsoft-Exchange-Diagnostics: 1;BY2PR15MB0407;31:KCCPTb3avVsR6V9MM43L2ARriIO6HXdYf3SZnwqiqbnvw0qW6BMzPh7YkyuOCIJ2X8a/2Kr8Pxjqvqek0RSnfrcL2gnOrvRxtHBmd8sHi+fhonVn6LiODlFGA5Rl8QL/XuNkJ0kWjRzbf1j28xDDm/BDe6NKstwMYljeKRh9YmJOpu6MIzSGNrjPkiP1DbxwzRZi30ib+n2veUsdOxnS0hbecYlm/ahF8z1GhUuzHKVadyjFWZMzhq5Mr2yZXIC+;20:4i8IwU6RFP/Hp0NJ8mEsSNd+yAdfgANwQUYupSEhL1c5/cyko/R+s1BRBSRVpBcyLqND+BKlV2TQQ4khxdI+luuSDzTu73gIm9R7zehZMVuTn6P+VKkmN/3GOMXT8bRaQ+lfuiwXvT6s8ax8c5h/UkXdguC4nSW7XpqZIX9/yY8=;4:ujnByR65M6hZumIuk61HqACQwUFB3uQt1DAcOhZlc/5/yAE0serTSkAECbO2OBZjWxocSWL+RdpFx29iS2mD8ArpdTcy7rIs10IjXcxBO+UTvBP24k0LUVcjVWFeEZPWKRhmYpEQcGeG4LIx9dj22AI5u2BiBYEtsrA+6ZlOXAqfVJE2hbqqrZu4pHtIlg0IV7pdUrL1w7so58rnlhqRIbfzK9X+yysa24aJyO4GK5CY1CADU3F8wjvJNJS+VCtg45jBgvmvxVZh/RjN4uVUo74VDdxePbtZZqjPdza3kPCJdPNtGAzHVQH6prZAdRcR9u2yGTHG01cDB/Lz0cMeQM8o48QztoVcWp88daYyW8Ciob3zBxp7qZmPSLr+426l71XCIUajoRxPRNr6jIyhqKULOy1eqwyeyheWpOs2fpnTg4ba+elDXv8fo1mrlL/3EMHWNp5RSkLETUykyWWZZg== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001);SRVR:BY2PR15MB0407;BCL:0;PCL:0;RULEID:;SRVR:BY2PR15MB0407; X-Forefront-PRVS: 00851CA28B X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(7916002)(189002)(199003)(24454002)(2906002)(4326007)(4001350100001)(83506001)(50466002)(6116002)(98436002)(23726003)(110136003)(97736004)(1076002)(189998001)(586003)(7736002)(33656002)(305945005)(7846002)(5660300001)(46406003)(92566002)(9686002)(8676002)(86362001)(54356999)(76176999)(81156014)(81166006)(105586002)(50986999)(19580405001)(19580395003)(77096005)(6666003)(97756001)(101416001)(42186005)(93886004)(106356001)(68736007)(6916009)(47776003)(2950100002)(18370500001)(142923001)(3826002);DIR:OUT;SFP:1102;SCL:1;SRVR:BY2PR15MB0407;H:anikkar-mbp.local.dhcp.thefacebook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BY2PR15MB0407;23:BwvaMKosrfaXlCOpAwlVv3sFjmbdLJ2nBAbKp/i8l?= =?us-ascii?Q?XmzaEBM8wkvf/XvKkc1tFGjOOnsuVxslzMTmclr3StLiv0DOSZh0DdJFVtoE?= =?us-ascii?Q?Zlj/l4pr2Twxk8iexFkKv7xhi7sQ6Fzu2dEqGsb4HfLuPAq1UKDJpFZDAFc/?= =?us-ascii?Q?F3omYch4MBHGM/KxhKO2rHxKKOq0vaYoLV7yYZLoyAzJy9Pm7CczWUzJUTK8?= =?us-ascii?Q?JlCmeBZ/byKsrqtKwP/+h0QElw0TzweNvWDnAFwRboOMXX5UyLv2o+b2+L6N?= =?us-ascii?Q?xBWwrJCIqsbZuaLdoSGM/XDQwHFh845C2GS8C3l63qBGFBQMxZ4Wde2D4qtg?= =?us-ascii?Q?SIKIsX0D2jnZLij3M10cD75BDgYvqmGN4WSFBXQKl70VRhc4TJVBVNX/ceJw?= =?us-ascii?Q?I8YnNMHzAdgBN5Fq2RM2VMx1EjDmcENWUZMNoNXVq1ZEkqcI3Yq/yq/YkSlU?= =?us-ascii?Q?6zqZBumZhhnS6LFyRFP+J0B8DijVMrjTE06yw7n/fd/4Ptm7K34oTJqG+gLc?= =?us-ascii?Q?Aw9u6lk/7KZlcczKMvAg8SHxJh3bVn4fuW8fVPjfHECH4zxOnxc1manUfOYT?= =?us-ascii?Q?ahhu8vHkXjTkMqEUOAhQRH/S/SCUAvdEtK5HNPjeVTiaLuaD5JR0GpRYe+wR?= =?us-ascii?Q?LI6F+qXRAQCdGb3HYVN6JeX8NNWGAhSnaQX1HWDwbg9J1sk2IOESw6xASEN0?= =?us-ascii?Q?HjCmD7s5DPeiY+ShnekbjO1diXuH8/1SBK5m/nwrI7kx7xyobiMUF/5gX5xm?= =?us-ascii?Q?aH96/igJdOa1b5Otu+1hvI+KV8T6WfDdKSLS9iLKPOz0z3T0U3Vkb3P4MsTP?= =?us-ascii?Q?9c7UshQMRrLKxCurT13by+Kh5Honlf/aI6zucl0zbLHeZldw919PcYLOVT1Z?= =?us-ascii?Q?mSrDFdlTuaILhjYSJLqtRcnAICCLBE1qUzVkoPspGeONDEGvqwFeoDNq5s1b?= =?us-ascii?Q?ON/FRZ3nBguA+ujtLX7nzPCNmY/oYto4R/EuIknA5Cxj7YmWDzH5unN8GtJZ?= =?us-ascii?Q?aH/lcaj4c8Z+MaLWXa3jxETi8WAV94Zx/kOfRlv8sAeLdG1KQbVUobfKd7Xb?= =?us-ascii?Q?3RbOgab2u8NZfBPYdm5B1rI5Y/NF5Xps7DIlPkeksvxGDommGiWzL5Jk0ADJ?= =?us-ascii?Q?RftcG5zpApM72KbhTv6f1TvHOZKMvWKnONxu8Pmq85eA2eeHqhiXG8LYuqkA?= =?us-ascii?Q?qNemoJqnOLK2doaOo1NTSmPlpg4P/W/nUpobeSBQxYYqk+UvTMqScC5SzYs8?= =?us-ascii?Q?YJ1ZDzef1jJWY+Dj6rayO9nyLP79rr9l80if9RZ?= X-Microsoft-Exchange-Diagnostics: 1;BY2PR15MB0407;6:SiuMIbafU4RMGL/oLp2IzYqkal/5AsDBHoOzUpN7p95U3wlHz+EhbGXu11Dcl12jFLHgVbf+F/zZ4d00mlKwnRGeqBMN9Ar+M0YSh9ZLaB4YfAEh8qew9Jisi8XmcWYHUrbtwiJFpdzSxet7/lh4J7Sc41oIMY1sS+//w7KW9d79Q8GxgtyIELWp2PWLBMfvEt7vU3a4e38tFWVeCm5o0ejo64QWTIgmCK/LJpOZIbJdcq9sWKpEwUDqJXffzo6JuI06DargnrqC22lfC+cNJThGboTyC7YAUiC/8pkz/bt082kkf4GgMeOxpoBy5eHz;5:GOZSdevX9tiTq6qMrJfp1W3caOOqSSJW8+Ps6cl8nrDuAynD5MMuIzNlpNiG4x9pQYUHGExumpZXx5HJL4MMfCmSioiMjlrEzJjU9pkyK5P+L9+XC5KmvehsvSV44vh2wLoLF2vpJ/5PK1MN4zG1O2Wu9KSIsMPnTx+g2HyMPQ0=;24:xkapmYTGFSwjs8O+/l4jnKCQh95x8Z3RCYD3oonM9CYMQ3ErTFw8StIaULi89kibcv/2HH1AHa+EgWu81nbRh6qxmUOm4IE473/HNBVuQfs=;7:lXSJ/XI/LMyOP7gpenbMqzCg6FxcWpbJdPCxVmegNsfgQspBxOJAVjz3BqnvjX8oxfC8A7cg4y3GGiqkfd7hFj7tCcVX3pAWMCUpIfPSdUN/latgQIk6W+EqSVv7zmaLtqZxHtEN7FDiqCe2aJy82k51nvrB8yLk8DqMTDJC+4ykqa94fmZpFEYcyEBenuWktEnIx1EmUQZBA3t9/0jLdMg6nBPq5nruZHUWw//b8fKa6FvXyFyMJZj/pYubLwRSB+554SFkHQG9QEysbTJuNmgecFOOo03rGMg601wnM1csG+iVfj7YPVURv9fXdWv6CYvaSdncv0DbZGRf0VPKWw== SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BY2PR15MB0407;20:UF2g/hFCWMQTyR6EgQ8VPABEZsmvmG2z5Dsoah82+yUdy2/tlw66f5QYBUnINt4RmuocKMAw0mKXIzk/5IcSiOtBDGywN/vV7lOvnWDyvRpVFzijO9CxX9nN1mX1+iPqDXrMgSdk9etKKeNwdZEXDVUDkDLXr2TZUg2sWL4OTHk= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Oct 2016 17:29:03.8646 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR15MB0407 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-04_06:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3937 Lines: 87 On Tue, Oct 04, 2016 at 07:01:39PM +0200, Paolo Valente wrote: > > > Il giorno 04 ott 2016, alle ore 18:27, Tejun Heo ha scritto: > > > > Hello, > > > > On Tue, Oct 04, 2016 at 06:22:28PM +0200, Paolo Valente wrote: > >> Could you please elaborate more on this point? BFQ uses sectors > >> served to measure service, and, on the all the fast devices on which > >> we have tested it, it accurately distributes > >> bandwidth as desired, redistributes excess bandwidth with any issue, > >> and guarantees high responsiveness and low latency at application and > >> system level (e.g., ~0 drop rate in video playback, with any background > >> workload tested). > > > > The same argument as before. Bandwidth is a very bad measure of IO > > resources spent. For specific use cases (like desktop or whatever), > > this can work but not generally. > > > > Actually, we have already discussed this point, and IMHO the arguments > that (apparently) convinced you that bandwidth is the most relevant > service guarantee for I/O in desktops and the like, prove that > bandwidth is the most important service guarantee in servers too. > > Again, all the examples I can think of seem to confirm it: > . file hosting: a good service must guarantee reasonable read/write, > i.e., download/upload, speeds to users > . file streaming: a good service must guarantee low drop rates, and > this can be guaranteed only by guaranteeing bandwidth and latency > . web hosting: high bandwidth and low latency needed here too > . clouds: high bw and low latency needed to let, e.g., users of VMs > enjoy high responsiveness and, for example, reasonable file-copy > time > ... > > To put in yet another way, with packet I/O in, e.g., clouds, there are > basically the same issues, and the main goal is again guaranteeing > bandwidth and low latency among nodes. > > Could you please provide a concrete server example (assuming we still > agree about desktops), where I/O bandwidth does not matter while time > does? I don't think IO bandwidth does not matter. The problem is bandwidth can't measure IO cost. For example, you can't say 8k IO costs 2x IO resource than 4k IO. > >> Could you please suggest me some test to show how sector-based > >> guarantees fails? > > > > Well, mix 4k random and sequential workloads and try to distribute the > > acteual IO resources. > > > > > If I'm not mistaken, we have already gone through this example too, > and I thought we agreed on what service scheme worked best, again > focusing only on desktops. To make a long story short(er), here is a > snippet from one of our last exchanges. > > ---------- > > On Sat, Apr 16, 2016 at 12:08:44AM +0200, Paolo Valente wrote: > > Maybe the source of confusion is the fact that a simple sector-based, > > proportional share scheduler always distributes total bandwidth > > according to weights. The catch is the additional BFQ rule: random > > workloads get only time isolation, and are charged for full budgets, > > so as to not affect the schedule of quasi-sequential workloads. So, > > the correct claim for BFQ is that it distributes total bandwidth > > according to weights (only) when all competing workloads are > > quasi-sequential. If some workloads are random, then these workloads > > are just time scheduled. This does break proportional-share bandwidth > > distribution with mixed workloads, but, much more importantly, saves > > both total throughput and individual bandwidths of quasi-sequential > > workloads. > > > > We could then check whether I did succeed in tuning timeouts and > > budgets so as to achieve the best tradeoffs. But this is probably a > > second-order problem as of now. I don't see why random/sequential matters for SSD. what really matters is request size and IO depth. Time scheduling is skeptical too, as workloads can dispatch all IO within almost 0 time in high queue depth disks. Thanks, Shaohua