Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 25 Oct 2021 12:05:27 -0400
From:   Kent Overstreet <kent.overstreet@gmail.com>
To:     Johannes Weiner <hannes@cmpxchg.org>
Cc:     Matthew Wilcox <willy@infradead.org>,
        "Kirill A. Shutemov" <kirill@shutemov.name>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        linux-mm@kvack.org, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Andrew Morton <akpm@linux-foundation.org>,
        "Darrick J. Wong" <djwong@kernel.org>,
        Christoph Hellwig <hch@infradead.org>,
        David Howells <dhowells@redhat.com>,
        Hugh Dickins <hughd@google.com>
Subject: Re: Folios for 5.15 request - Was: re: Folio discussion recap -
Message-ID: <YXbVxwx+Ln73CtEB@moria.home.lan>
References: <YWpG1xlPbm7Jpf2b@casper.infradead.org>
 <YW2lKcqwBZGDCz6T@cmpxchg.org>
 <YW28vaoW7qNeX3GP@casper.infradead.org>
 <YW3tkuCUPVICvMBX@cmpxchg.org>
 <20211018231627.kqrnalsi74bgpoxu@box.shutemov.name>
 <YW7hQlny+Go1K3LT@cmpxchg.org>
 <YXBUPguecSeSO6UD@moria.home.lan>
 <YXHdpQTL1Udz48fc@cmpxchg.org>
 <YXIZX0truEBv2YSz@casper.infradead.org>
 <YXbOvR6jMXZ0WPcM@cmpxchg.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YXbOvR6jMXZ0WPcM@cmpxchg.org>
Precedence: bulk

On Mon, Oct 25, 2021 at 11:35:25AM -0400, Johannes Weiner wrote:
> On Fri, Oct 22, 2021 at 02:52:31AM +0100, Matthew Wilcox wrote:
> > > Anyway. I can even be convinved that we can figure out the exact fault
> > > lines along which we split the page down the road.
> > > 
> > > My worry is more about 2). A shared type and generic code is likely to
> > > emerge regardless of how we split it. Think about it, the only world
> > > in which that isn't true would be one in which either
> > > 
> > > 	a) page subtypes are all the same, or
> > > 	b) the subtypes have nothing in common
> > > 
> > > and both are clearly bogus.
> > 
> > Amen!
> > 
> > I'm convinced that pgtable, slab and zsmalloc uses of struct page can all
> > be split out into their own types instead of being folios.  They have
> > little-to-nothing in common with anon+file; they can't be mapped into
> > userspace and they can't be on the LRU.  The only situation you can find
> > them in is something like compaction which walks PFNs.
> 
> They can all be accounted to a cgroup. pgtables are tracked the same
> as other __GFP_ACCOUNT pages (pipe buffers and kernel stacks right now
> from a quick grep, but as you can guess that's open-ended).
> 
> So if those all aren't folios, the generic type and the interfacing
> object for memcg and accounting would continue to be the page.
> 
> > Perhaps you could comment on how you'd see separate anon_mem and
> > file_mem types working for the memcg code?  Would you want to have
> > separate lock_anon_memcg() and lock_file_memcg(), or would you want
> > them to be cast to a common type like lock_folio_memcg()?
> 
> That should be lock_<generic>_memcg() since it actually serializes and
> protects the same thing for all subtypes (unlike lock_page()!).
> 
> The memcg interface is fully type agnostic nowadays, but it also needs
> to be able to handle any subtype. It should continue to interface with
> the broadest, most generic definition of "chunk of memory".
> 
> Notably it does not do tailpages (and I don't see how it ever would),
> so it could in theory use the folio - but only if the folio is really
> the systematic replacement of absolutely *everything* that isn't a
> tailpage - including pgtables, kernel stack, pipe buffers, and all
> other random alloc_page() calls spread throughout the code base. Not
> just conceptually, but an actual wholesale replacement of struct page
> throughout allocation sites.
> 
> I'm not sure that's realistic. So I'm thinking struct page will likely
> be the interfacing object for memcg for the foreseeable future.

Interesting.

We were also just discussing how in the block layer, bvecs can currently point
to multiple pages - this is the multipage bvec work that Ming did, it made bio
segment merging a lot cheaper by moving it from the layer that maps bvecs to
sglists and up to bio_add_page() and got rid of the need for segment counting.

But with the upper layers transitioning to compound pages, i.e. keeping
contiguous stuff together as a unit - we're going to want to switch bvecs to
pointing to compound pages, and ditch all the code that breaks up a bvec into
individual 4k pages when we iterate over them; we also won't need or want any
kind of page/segment merging anymore, which is really cool.

But since bios can do IO to/from basically any type of memory, this is another
argument in favor of folios becoming the replacement for all or essentially all
compound pages. The alternative would be changing bvecs to only point to head
pages, which I do think would be completely workable with appropriate
assertions.

We don't want to prevent doing block IO to/from slab memory - there's a lot of
places where we do block IO to memory that isn't exposed to userspace
(e.g. filesystem metadata, other weirder paths), so if bvecs point to folios,
then at least slab needs to be a subtype of folios and folios need to be all or
most compound pages.

I've been anti folios being the replacement for all compound pages because this
is C, trying to do a lot with types is a pain in the ass and I think in general
nested inheritence heirarchies tend to not be the way to go. But I'm definitely
keeping an open mind...