Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752892AbYKQOZ4 (ORCPT ); Mon, 17 Nov 2008 09:25:56 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752213AbYKQOZr (ORCPT ); Mon, 17 Nov 2008 09:25:47 -0500 Received: from mx2.redhat.com ([66.187.237.31]:49066 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752157AbYKQOZq (ORCPT ); Mon, 17 Nov 2008 09:25:46 -0500 Date: Mon, 17 Nov 2008 09:23:09 -0500 From: Vivek Goyal To: Nauman Rafique Cc: Divyesh Shah , Ryo Tsuruta , linux-kernel@vger.kernel.org, containers@lists.linux-foundation.org, virtualization@lists.linux-foundation.org, jens.axboe@oracle.com, taka@valinux.co.jp, righi.andrea@gmail.com, s-uchida@ap.jp.nec.com, fernando@oss.ntt.co.jp, balbir@linux.vnet.ibm.com, akpm@linux-foundation.org, menage@google.com, ngupta@google.com, riel@redhat.com, jmoyer@redhat.com, peterz@infradead.org, Fabio Checconi , paolo.valente@unimore.it Subject: Re: [patch 0/4] [RFC] Another proportional weight IO controller Message-ID: <20081117142309.GA15564@redhat.com> References: <20081106153022.215696930@redhat.com> <20081113.180558.519459540419535699.ryov@valinux.co.jp> <20081113155834.GE7542@redhat.com> <20081113214642.GG7542@redhat.com> <20081114160525.GE24624@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5901 Lines: 121 On Fri, Nov 14, 2008 at 02:44:22PM -0800, Nauman Rafique wrote: > In an attempt to make sure that this discussion leads to > something useful, we have summarized the points raised in this > discussion and have come up with a strategy for future. > The goal of this is to find common ground between all the approaches > proposed on this mailing list. > > 1 Start with Satoshi's latest patches. I have had a brief look at both Satoshi's patch and bfq. I kind of like bfq's patches for keeping track of per cgroup, per queue data structures. May be we can look there also. > 2 Do the following to support propotional division: > a) Give time slices in proportion to weights (configurable > option). We can support both priorities and weights by doing > propotional division between requests with same priorities. > 3 Schedule time slices using WF2Q+ instead of round robin. > Test the performance impact (both throughput and jitter in latency). > 4 Do the following to support the goals of 2 level schedulers: > a) Limit the request descriptors allocated to each cgroup by adding > functionality to elv_may_queue() > b) Add support for putting an absolute limit on IO consumed by a > cgroup. Such support exists in dm-ioband and is provided by Andrea > Righi's patches too. Does dm-iobnd support abosolute limit? I think till last version they did not. I have not check the latest version though. > c) Add support (configurable option) to keep track of total disk > time/sectors/count > consumed at each device, and factor that into scheduling decision > (more discussion needed here) > 5 Support multiple layers of cgroups to align IO controller behavior > with CPU scheduling behavior (more discussion?) > 6 Incorporate an IO tracking approach which re-uses memory resource > controller code but is not dependent on it (may be biocgroup patches from > dm-ioband can be used here directly) > 7 Start an offline email thread to keep track of progress on the above > goals. > > Please feel free to add/modify items to the list > when you respond back. Any comments/suggestions are more than welcome. > Thanks Vivek > Thanks. > Divyesh & Nauman > > On Fri, Nov 14, 2008 at 8:05 AM, Vivek Goyal wrote: > > On Thu, Nov 13, 2008 at 02:57:29PM -0800, Divyesh Shah wrote: > > > > [..] > >> > > > Ryo, do you still want to stick to two level scheduling? Given the problem > >> > > > of it breaking down underlying scheduler's assumptions, probably it makes > >> > > > more sense to the IO control at each individual IO scheduler. > >> > > > >> > > Vivek, > >> > > I agree with you that 2 layer scheduler *might* invalidate some > >> > > IO scheduler assumptions (though some testing might help here to > >> > > confirm that). However, one big concern I have with proportional > >> > > division at the IO scheduler level is that there is no means of doing > >> > > admission control at the request queue for the device. What we need is > >> > > request queue partitioning per cgroup. > >> > > Consider that I want to divide my disk's bandwidth among 3 > >> > > cgroups(A, B and C) equally. But say some tasks in the cgroup A flood > >> > > the disk with IO requests and completely use up all of the requests in > >> > > the rq resulting in the following IOs to be blocked on a slot getting > >> > > empty in the rq thus affecting their overall latency. One might argue > >> > > that over the long term though we'll get equal bandwidth division > >> > > between these cgroups. But now consider that cgroup A has tasks that > >> > > always storm the disk with large number of IOs which can be a problem > >> > > for other cgroups. > >> > > This actually becomes an even larger problem when we want to > >> > > support high priority requests as they may get blocked behind other > >> > > lower priority requests which have used up all the available requests > >> > > in the rq. With request queue division we can achieve this easily by > >> > > having tasks requiring high priority IO belong to a different cgroup. > >> > > dm-ioband and any other 2-level scheduler can do this easily. > >> > > > >> > > >> > Hi Divyesh, > >> > > >> > I understand that request descriptors can be a bottleneck here. But that > >> > should be an issue even today with CFQ where a low priority process > >> > consume lots of request descriptors and prevent higher priority process > >> > from submitting the request. > >> > >> Yes that is true and that is one of the main reasons why I would lean > >> towards 2-level scheduler coz you get request queue division as well. > >> > >> I think you already said it and I just > >> > reiterated it. > >> > > >> > I think in that case we need to do something about request descriptor > >> > allocation instead of relying on 2nd level of IO scheduler. > >> > At this point I am not sure what to do. May be we can take feedback from the > >> > respective queue (like cfqq) of submitting application and if it is already > >> > backlogged beyond a certain limit, then we can put that application to sleep > >> > and stop it from consuming excessive amount of request descriptors > >> > (despite the fact that we have free request descriptors). > >> > >> This should be done per-cgroup rather than per-process. > >> > > > > Yep, per cgroup limit will make more sense. get_request() already calls > > elv_may_queue() to get a feedback from IO scheduler. May be here IO > > scheduler can make a decision how many request descriptors are already > > allocated to this cgroup. And if the queue is congested, then IO scheduler > > can deny the fresh request allocation. > > > > Thanks > > Vivek > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/