Received-SPF: pass (google.com: domain of linux-kernel+bounces-31239-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Precedence: bulk
MIME-Version: 1.0
References: <20240118120347.61817-1-ioworker0@gmail.com> <ZakqQyL9t2ffNUIf@tiehlicka>
 <CAK1f24k+=Sskotbct+yGxpDKNv=qyXPkww5i2kaqfzwaUVO_GQ@mail.gmail.com> <ZapwWuVTIDeI3W8A@tiehlicka>
In-Reply-To: <ZapwWuVTIDeI3W8A@tiehlicka>
From: Lance Yang <ioworker0@gmail.com>
Date: Fri, 19 Jan 2024 22:08:44 +0800
Message-ID: <CAK1f24=eVDNRN3k4Oz62VZ4M=1igVQ9E-0vxDe=5M8HWrAb-8Q@mail.gmail.com>
Subject: Re: [PATCH v2 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT to process_madvise()
To: Michal Hocko <mhocko@suse.com>
Cc: akpm@linux-foundation.org, zokeefe@google.com, david@redhat.com, 
	songmuchun@bytedance.com, shy828301@gmail.com, peterx@redhat.com, 
	mknyszek@google.com, minchan@kernel.org, linux-mm@kvack.org, 
	linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Jan 19, 2024 at 8:51=E2=80=AFPM Michal Hocko <mhocko@suse.com> wrot=
e:
>
> On Fri 19-01-24 10:03:05, Lance Yang wrote:
> > Hey Michal,
> >
> > Thanks for taking the time to review!
> >
> > On Thu, Jan 18, 2024 at 9:40=E2=80=AFPM Michal Hocko <mhocko@suse.com> =
wrote:
> > >
> > > On Thu 18-01-24 20:03:46, Lance Yang wrote:
> > > [...]
> > >
> > > before we discuss the semantic, let's focus on the usecase.
> > >
> > > > Use Cases
> > > >
> > > > An immediate user of this new functionality is the Go runtime heap =
allocator
> > > > that manages memory in hugepage-sized chunks. In the past, whether =
it was a
> > > > newly allocated chunk through mmap() or a reused chunk released by
> > > > madvise(MADV_DONTNEED), the allocator attempted to eagerly back mem=
ory with
> > > > huge pages using madvise(MADV_HUGEPAGE)[2] and madvise(MADV_COLLAPS=
E)[3]
> > > > respectively. However, both approaches resulted in performance issu=
es; for
> > > > both scenarios, there could be entries into direct reclaim and/or c=
ompaction,
> > > > leading to unpredictable stalls[4]. Now, the allocator can confiden=
tly use
> > > > process_madvise(MADV_F_COLLAPSE_LIGHT) to attempt the allocation of=
 huge pages.
> > >
> > > IIUC the primary reason is the cost of the huge page allocation which
> > > can be really high if the memory is heavily fragmented and it is call=
ed
> > > synchronously from the process directly, correct? Can that be worked
> >
> > Yes, that's correct.
> >
> > > around by process_madvise and performing the operation from a differe=
nt
> > > context? Are there any other reasons to have a different mode?
> >
> > In latency-sensitive scenarios, some applications aim to enhance perfor=
mance
> > by utilizing huge pages as much as possible. At the same time, in case =
of
> > allocation failure, they prefer a quick return without triggering direc=
t memory
> > reclamation and compaction.
>
> Could you elaborate some more on why?

Previously, the Go runtime attempted to marks all new memory as MADV_HUGEPA=
GE
on Linux and manages its hugepage eligibility status. Unfortunately,
the default THP
behavior on most Linux distros is that MADV_HUGEPAGE blocks while the kerne=
l
eagerly reclaims and compacts memory to allocate a hugepage.
This direct reclaim and compaction is unbounded, and may result in signific=
ant
application thread stalls. In really bad cases, this can exceed 100s
of ms or even
seconds.
The overall strategy of trying to keep hugepages for the heap unbroken
however is
sound. So, the Go runtime uses MADV_COLLAPSE as an alternative.
See https://github.com/golang/go/commit/9f9bb26880388c5bead158e9eca3be4b3a9=
bd2af

Later, a Google production service experienced a performance
regression with the Go
runtime's use of MADV_COLLAPSE. For now, the Go runtime has rolled
back the usage of MADV_COLLAPSE.
See https://github.com/golang/go/issues/63334

If there were a more relaxed (opportunistic) MADV_COLLAPSE, it would
avoid direct reclaim
and/or compaction and quickly fail on allocation errors. This could be
beneficial for similar
use cases.

BR,
Lance

>
> > > I mean I can think of a more relaxed (opportunistic) MADV_COLLAPSE -
> > > e.g. non blocking one to make sure that the caller doesn't really blo=
ck
> > > on resource contention (be it locks or memory availability) because t=
hat
> > > matches our non-blocking interface in other areas but having a LIGHT
> > > operation sounds really vague and the exact semantic would be
> > > implementation specific and might change over time. Non-blocking has =
a
> > > clear semantic but it is not really clear whether that is what you
> > > really need/want.
> >
> > Could you provide me with some suggestions regarding the naming of a
> > more relaxed (opportunistic) MADV_COLLAPSE?
>
> Naming is not all that important at this stage (it could be
> MADV_COLLAPSE_NOBLOCK for example). The primary question is whether
> non-blocking in general is the desired behavior or the implementation
> should try but not too hard.
>
> --
> Michal Hocko
> SUSE Labs