Received-SPF: pass (google.com: domain of linux-kernel+bounces-31678-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <20240117050217.43610-1-ioworker0@gmail.com> <87ededl9u3.fsf@linux.intel.com>
In-Reply-To: <87ededl9u3.fsf@linux.intel.com>
From: Lance Yang <ioworker0@gmail.com>
Date: Sat, 20 Jan 2024 10:34:37 +0800
Message-ID: <CAK1f24=cG4S2MzdMuZOvSmzrj0QFSXTeuVpfuObhUNGxDMSKWg@mail.gmail.com>
Subject: Re: [PATCH v1 1/2] mm/madvise: introduce MADV_TRY_COLLAPSE for
 attempted synchronous hugepage collapse
To: Andi Kleen <ak@linux.intel.com>
Cc: akpm@linux-foundation.org, zokeefe@google.com, songmuchun@bytedance.com, 
	linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey Andi,

Thanks for taking the time to review!

We are currently discussing this at
Link: https://lore.kernel.org/all/20240118120347.61817-1-ioworker0@gmail.co=
m/

On Fri, Jan 19, 2024 at 9:41=E2=80=AFPM Andi Kleen <ak@linux.intel.com> wro=
te:
>
> Lance Yang <ioworker0@gmail.com> writes:
>
> > This idea was inspired by MADV_COLLAPSE introduced by Zach O'Keefe[1].
> >
> > Introduce a new madvise mode, MADV_TRY_COLLAPSE, that allows users to
> > make a least-effort attempt at a synchronous collapse of memory at
> > their own expense.
> >
> > The only difference from MADV_COLLAPSE is that the new hugepage allocat=
ion
> > avoids direct reclaim and/or compaction, quickly failing on allocation =
errors.
> >
> > The benefits of this approach are:
> >
> > * CPU is charged to the process that wants to spend the cycles for the =
THP
> > * Avoid unpredictable timing of khugepaged collapse
> > * Prevent unpredictable stalls caused by direct reclaim and/or
> > compaction
>
> I haven't completely followed the discussion, but it seem your second
> and third point could be addressed by a asynchronous THP fault without
> any new APIs: allocate 2MB while failing quickly, then on failure get
> a 4K page and provide it to the process, while asking khugepaged to
> convert the page ASAP in the background, but only after
> it managed to allocate a fresh 2MB page to minimize the process visible
> down time.
>
> I suppose that would be much more predictable, although there would be a
> slightly risk of overwhelming khugepaged. The later could be
> addressed by using a scalable workqueue that allocates more threads
> when needed.

Thank you for your suggestion!

Unfortunately, AFAIK, the default THP behavior on most Linux distros is
that MADV_HUGEPAGE blocks while the kernel eagerly reclaims
and compacts memory to allocate a hugepage.

In the era of cloud-native computing, it's challenging for users to be
aware of the THP configurations on all nodes in a cluster, let alone
have fine-grained control over them. Simply disabling the use of huge
pages due to concerns about potential direct reclaim and/or compaction
may be regrettable, as users are deprived of the opportunity to experiment
with huge page allocations. However, relying solely on
MADV_HUGEPAGE introduces the risk of unpredictable stalls,
making it a trade-off that users must carefully consider.

With the introduction of MADV_COLLAPSE into the kernel, it
is not governed by the defrag mode. MADV_COLLAPSE
offers the potential for more fine-grained control over the hugepage
allocation strategy.

BR,
Lance
>
> -Andi
>