Received-SPF: pass (google.com: domain of linux-kernel+bounces-40869-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Precedence: bulk
MIME-Version: 1.0
References: <20240118120347.61817-1-ioworker0@gmail.com> <CAK1f24=TfJvsDCEesaTa8rGP7ay62p6UiJem=XWnpFa9yfSA3A@mail.gmail.com>
 <CAK1f24nS8MEA3wcS4za-uSp7ZBxvd+xqMRf8-u=m5uCvTs8yJQ@mail.gmail.com>
 <CAK1f24kOYDOw26ov5TVpAyNP13hCjm=cDo4rooOTPDuv8L6Pnw@mail.gmail.com> <CAK1f24n0152CmPghBLQ7225=rwEuP6mMdBybWZz7heTLzemBqw@mail.gmail.com>
In-Reply-To: <CAK1f24n0152CmPghBLQ7225=rwEuP6mMdBybWZz7heTLzemBqw@mail.gmail.com>
From: "Zach O'Keefe" <zokeefe@google.com>
Date: Fri, 26 Jan 2024 15:46:21 -0800
Message-ID: <CAAa6QmRZP5pL_5O7BpfjQf5LZ_ADGqYF_xdAYEbKXkqMViAwLw@mail.gmail.com>
Subject: Re: [PATCH v2 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT to process_madvise()
To: Lance Yang <ioworker0@gmail.com>
Cc: akpm@linux-foundation.org, Michal Hocko <mhocko@suse.com>, 
	Yang Shi <shy828301@gmail.com>, David Hildenbrand <david@redhat.com>, songmuchun@bytedance.com, 
	peterx@redhat.com, mknyszek@google.com, minchan@kernel.org, 
	linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

> I=E2=80=99d like to add another real use case.
>
> In our company, we deploy applications using offline-online
> hybrid deployment. This approach leverages the distinctive
> resource utilization patterns of online services, utilizing idle
> resources during various time periods by filling them with
> offline jobs. This helps reduce the growing cost expenditures
> for the enterprise.
>
> Whether for online services or offline jobs, their requirements
> for THP can be roughly categorized into three types:
>
> * The first type aims to use huge pages as much as possible
> and tolerates unpredictable stalls caused by direct reclaim
> and/or compaction.
> * The second type attempts to use huge pages but is relatively
> latency-sensitive and cannot tolerate unpredictable stalls.
> * The third type prefers not to use huge pages at all and is
> extremely latency-sensitive.
>
> After careful consideration, we decided to prioritize the
> requirements of the first type and modify the THP settings
> as follows:
>
> echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
> echo defer >/sys/kernel/mm/transparent_hugepage/defrag
>
> With the introduction of MADV_COLLAPSE into the kernel,
> it is no longer dependent on any sysfs setting under
> /sys/kernel/mm/transparent_hugepage. MADV_COLLAPSE
> offers the potential for fine-grained synchronous control over
> the huge page allocation mechanism, marking a significant
> enhancement for THP.
>
> If the kernel supports a more relaxed (opportunistic)
> MADV_COLLAPSE, we will modify the THP settings as follows:
>
> echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
> echo madvise >/sys/kernel/mm/transparent_hugepage/defrag

[corrected, via 2 previous mails, to: echo madvise
>/sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag]


> Then, we will use process_madvise(MADV_COLLAPSE, xx_relaxed_flag)
> to address the requirements of the second type.
>
> Why don't we favor madvise(MADV_COLLAPSE) for the first type
> of requirements?
> The main reason is that these requirements are typically for offline
> jobs in the Hadoop ecosystem, such as MapReduce and Spark,
> which run primarily on the JVM. [..]

Hey Lance,

Thanks for proving this context, it's very helpful.

Though, couldn't you use enabled=3Dalways, defrag=3Ddefer+madvise, then
just use prctl(PR_SET_THP_DISABLE) on type-3 workloads to get the
behaviour you want? i.e.

type 1: apply MADV_HUGEPAGE -> sync defrag to get THP
type 2: don't apply MADV_HUGEPAGE -> use THP if available, kick
kswapd+kcompactd otherwise
type 3: use prctl(PR_SET_THP_DISABLE) (or MADV_NOHUGEPAGE) -> no THPs

Or am I missing something? It sounds like a confounding issue is that
these are external workloads, or you don't have ability to modify? But
that would preclude MADV_COLLAPSE (unless you're using
process_madvise()).

Appreciate the help understanding the use case. I'm not opposed to the
idea in general, but IMO would be great to have a clear need for it
(and right now, we don't currently have alignment with the original
motivating usecase (Go) in that regard w.r.t their plans).

Thanks,
Zach