Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754943Ab1CQSQQ (ORCPT ); Thu, 17 Mar 2011 14:16:16 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57069 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753529Ab1CQSQN (ORCPT ); Thu, 17 Mar 2011 14:16:13 -0400 Date: Thu, 17 Mar 2011 14:15:43 -0400 From: Vivek Goyal To: Jan Kara Cc: Greg Thelen , Johannes Weiner , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, containers@lists.osdl.org, linux-fsdevel@vger.kernel.org, Andrea Righi , Balbir Singh , KAMEZAWA Hiroyuki , Daisuke Nishimura , Minchan Kim , Ciju Rajan K , David Rientjes , Wu Fengguang , Chad Talbott , Justin TerAvest , Curt Wohlgemuth Subject: Re: [PATCH v6 0/9] memcg: per cgroup dirty page accounting Message-ID: <20110317181543.GA10482@redhat.com> References: <20110314202324.GG31120@redhat.com> <20110315184839.GB5740@redhat.com> <20110316131324.GM2140@cmpxchg.org> <20110316215214.GO2140@cmpxchg.org> <20110317144641.GC4116@quack.suse.cz> <20110317171219.GD32392@redhat.com> <20110317175908.GH4116@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110317175908.GH4116@quack.suse.cz> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3903 Lines: 68 On Thu, Mar 17, 2011 at 06:59:08PM +0100, Jan Kara wrote: > On Thu 17-03-11 13:12:19, Vivek Goyal wrote: > > On Thu, Mar 17, 2011 at 03:46:41PM +0100, Jan Kara wrote: > > [..] > > > > - bdi writeback: will revert some of the mmotm memcg dirty limit changes to > > > > fs-writeback.c so that wb_do_writeback() will return to checking > > > > wb_check_background_flush() to check background limits and being > > > > interruptible if > > > > sync flush occurs. wb_check_background_flush() will check the global > > > > memcg_over_bg_limit list for memcg that are over their dirty limit. > > > > wb_writeback() will either (I am not sure): > > > > a) scan memcg's bdi_memcg list of inodes (only some of them are dirty) > > > > b) scan bdi dirty inode list (only some of them in memcg) using > > > > inode_in_memcg() to identify inodes to write. inode_in_memcg(inode,memcg), > > > > would walk memcg- -> memcg_bdi -> memcg_mapping to determine if the memcg > > > > is caching pages from the inode. > > > Hmm, both has its problems. With a) we could queue all the dirty inodes > > > from the memcg for writeback but then we'd essentially write all dirty data > > > for a memcg, not only enough data to get below bg limit. And if we started > > > skipping inodes when memcg(s) inode belongs to get below bg limit, we'd > > > risk copying inodes there and back without reason, cases where some inodes > > > never get written because they always end up skipped etc. Also the question > > > whether some of the memcgs inode belongs to is still over limit is the > > > hardest part of solution b) so we wouldn't help ourselves much. > > > > May be I am missing something but can't we just start traversing > > through list of memcg_over_bg_list and take option a) to traverse > > through list of inodes and write them till we are below limit of > > that group. We of course skip inodes which are not dirty. > > > > This is assuming that root group is also part of that list so that > > inodes in root group do not starve writeback. > > > > We still continue to have all the inodes on bdi wb structure and > > memcg will just give us pointers to those inodes. So for background > > write, instead of going serially through dirty inodes list, we > > will first pick the cgroup to write and then inode to write. As > > we will be doing round robin among cgroup list, it will make sure > > that none of the cgroups (including root) as well as inode are not > > starved. > I was considering this as well and didn't quite like it but on a second > thought it need not be that bad. If we wrote MAX_WRITEBACK_PAGES from one > memcg, then switched to another one while keeping pointers to per-memcg inode > list (for the time when we return to this memcg), it could work just fine. Yes, we can write MAX_WRITEBACK_PAGES from each memcg and then move on to next one. In fact memcg_bdi should have list of memcg_mapping. So once we select the inode (memcg_mapping) from cgroup for writeout (move inode on ->b_io list), we can also shuffle the position of memcg_mapping with-in memory cgroup so that inodes with-in a cgroup get fair share of writeout in a round robin manner. As you said in other mail, we probably will keep MEMCG_BDI_WRITTEN count so that IO less throttling can distribute the pages completed to right cgroup. Down the line we can probably also maintain MEMCG_BDI_WRITEBACK to keep track how many pages are already under writeout from a cgroup and skip that cgroup if too many pages are already in-flight. This might help us push more WRITES for higher weight IO cgroup as compared to lower weight IO cgroup. Thanks Vivek -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/