Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 24 Jan 2022 10:48:55 -0800 (PST)
From:   David Rientjes <rientjes@google.com>
To:     Peter Xu <peterx@redhat.com>, Zach O'Keefe <zokeefe@google.com>,
        SeongJae Park <sj@kernel.org>
cc:     Shakeel Butt <shakeelb@google.com>,
        David Hildenbrand <david@redhat.com>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Yang Shi <shy828301@gmail.com>, Zi Yan <ziy@nvidia.com>,
        Matthew Wilcox <willy@infradead.org>,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] mm: split thp synchronously on MADV_DONTNEED
In-Reply-To: <YaBevbuNuR+ToJ1o@xz-m1.local>
Message-ID: <e55d1f78-46c2-9ced-d7ad-c6deba4cb7b8@google.com>
References: <20211120201230.920082-1-shakeelb@google.com> <25b36a5c-5bbd-5423-0c67-05cd6c1432a7@redhat.com> <CALvZod5L1C1DV_DVs9O3xZm6CJnriunAoj89YLDdCp7ef5yBxA@mail.gmail.com> <YZ9kUD5AG6inbUEg@xz-m1.local> <CALvZod7L5Ga2xZOy_hgocsTxSuOYs840TiviOAhRwz59ATubWg@mail.gmail.com>
 <YaBevbuNuR+ToJ1o@xz-m1.local>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Fri, 26 Nov 2021, Peter Xu wrote:

> Some side notes: I digged out the old MADV_COLLAPSE proposal right after I
> thought about MADV_SPLIT (or any of its variance):
> 
> https://lore.kernel.org/all/d098c392-273a-36a4-1a29-59731cdf5d3d@google.com/
> 
> My memory was that there's some issue to be solved so that was blocked, however
> when I read the thread it sounds like the list was mostly reaching a consensus
> on considering MADV_COLLAPSE being beneficial.  Still copying DavidR in case I
> missed something important.
> 
> If we think MADV_COLLAPSE can help to implement an userspace (and more
> importantly, data-aware) khugepaged, then MADV_SPLIT can be the other side of
> kcompactd, perhaps.
> 
> That's probably a bit off topic of this specific discussion on the specific use
> case, but so far it seems all reasonable and discussable.
> 

Hi Peter,

Providing a (late) update since we now have some better traction on this, 
I think we'll be ready to post an RFC soon that introduces MADV_COLLAPSE.  
The work is being driven by Zach, now cc'd.

Let's also include SeongJae Park <sj@kernel.org> as well and keep him in 
the loop since DAMON could easily be extended with a DAMOS_COLLAPSE action 
to use MADV_COLLAPSE for hot regions of memory.

Idea for initial approach:

 - MADV_COLLAPSE core code based on the proposal you cite above for anon 
   memory as the inaugural support, collapse memory into thp in process 
   context

 - Batching support to collapse ranges of memory into multiple THP

 - Wire this up for madvise(2) (and process_madvise(2))

 - Enlightenment for file-backed thp

I think Zach's RFC will cover the first three, it could be debated if the 
initial patch series *must* support file-backed thp.  We'll see based on 
the feedback to the RFC.

There's also an extension where MADV_COLLAPSE could be potentially useful 
for hugetlb backed memory.  We have another effort underway that we've 
been talking with Mike Kravetz about that allows hugetlb memory to be 
mapped at multiple levels of the page tables.  There are several use cases 
but one of the driving factors is the performance of post-copy live 
migration; in this case, you'd be able to send smaller sized pages over 
the wire rather than, say, a 1GB gigantic page.

In this case, MADV_COLLAPSE could be useful to map smaller pages by 
a larger page table entry before all of the smaller pages have been live 
migrated.

That said, we have not invested time into an MADV_SPLIT yet.

Do you (or anybody else) have concerns about this approach?  Ideas for 
extensions?

Thanks!