Message-ID: <861f98b5-9211-98c7-b4f7-fd71146aa64c@redhat.com>
Date:   Mon, 22 Nov 2021 10:19:09 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.2.0
Content-Language: en-US
To:     Matthew Wilcox <willy@infradead.org>,
        Shakeel Butt <shakeelb@google.com>
Cc:     "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Yang Shi <shy828301@gmail.com>, Zi Yan <ziy@nvidia.com>,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org
References: <20211120201230.920082-1-shakeelb@google.com>
 <YZsi+RFed3hX9T8w@casper.infradead.org>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
Subject: Re: [PATCH] mm: split thp synchronously on MADV_DONTNEED
In-Reply-To: <YZsi+RFed3hX9T8w@casper.infradead.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 22.11.21 05:56, Matthew Wilcox wrote:
> On Sat, Nov 20, 2021 at 12:12:30PM -0800, Shakeel Butt wrote:
>> Many applications do sophisticated management of their heap memory for
>> better performance but with low cost. We have a bunch of such
>> applications running on our production and examples include caching and
>> data storage services. These applications keep their hot data on the
>> THPs for better performance and release the cold data through
>> MADV_DONTNEED to keep the memory cost low.
>>
>> The kernel defers the split and release of THPs until there is memory
>> pressure. This causes complicates the memory management of these
>> sophisticated applications which then needs to look into low level
>> kernel handling of THPs to better gauge their headroom for expansion. In
>> addition these applications are very latency sensitive and would prefer
>> to not face memory reclaim due to non-deterministic nature of reclaim.
>>
>> This patch let such applications not worry about the low level handling
>> of THPs in the kernel and splits the THPs synchronously on
>> MADV_DONTNEED.
> 
> I've been wondering about whether this is really the right strategy
> (and this goes wider than just this one, new case)
> 
> We chose to use a 2MB page here, based on whatever heuristics are
> currently in play.  Now userspace is telling us we were wrong and should
> have used smaller pages.

IIUC, not necessarily, unfortunately.

User space might be discarding the whole 2MB either via a single call
(MADV_DONTNEED a 2MB range as done by virtio-balloon with "free page
reporting" or by virtio-mem in QEMU). In that case, there is nothing to
migrate and we were not doing anything wrong.

But more extreme, user space might be discarding the whole THP in small
pieces shortly over time. This for example happens when a VM inflates
the memory balloon via virtio-balloon. All inflation requests are 4k,
resulting in a 4k MADV_DONTNEED calls. If we end up inflating a THP
range inside of the VM, mapping to a THP range inside the hypervisor,
we'll essentially free a THP in the hypervisor piece by piece using
individual MADV_DONTNEED calls -- this happens frequently. Something
similar can happen when de-fragmentation inside the VM "moves around"
inflated 4k pages piece by piece to essentially form a huge inflated
range -- this happens less frequently as of now. In both cases,
migration is counter-productive, as we're just about to free the whole
range either way.

(yes, there are ways to optimize, for example using hugepage ballooning
or merging MADV_DONTNEED calls in the hypervisor, but what I described
is what we currently implement in hypervisors like QEMU, because there
are corner cases for everything)

Long story short: it's hard to tell what will happen next based on a
single MADV_DONTNEED call. Page compaction, in comparison, doesn't care
and optimized the layout as it observes it.

-- 
Thanks,

David / dhildenb