MIME-Version: 1.0
References: <20230306235730.GA31451@monkey> <ZAaCISgq4A/GnkCk@x1n>
 <20230307004049.GC4956@monkey> <20230308190206.GA4005@monkey>
In-Reply-To: <20230308190206.GA4005@monkey>
From:   "Zach O'Keefe" <zokeefe@google.com>
Date:   Thu, 9 Mar 2023 14:38:01 -0800
Message-ID: <CAAa6QmSdTo77dP2q2nU==C_2PdyF611+PVF32uPOTUQbp1kc9Q@mail.gmail.com>
Subject: Re: THP backed thread stacks
To:     Mike Kravetz <mike.kravetz@oracle.com>
Cc:     Peter Xu <peterx@redhat.com>, David Hildenbrand <david@redhat.com>,
        Rik van Riel <riel@surriel.com>,
        Mike Rapoport <rppt@kernel.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, Mar 8, 2023 at 11:02=E2=80=AFAM Mike Kravetz <mike.kravetz@oracle.c=
om> wrote:
>
> On 03/06/23 16:40, Mike Kravetz wrote:
> > On 03/06/23 19:15, Peter Xu wrote:
> > > On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote:
> > > >
> > > > Just wondering if there is anything better or more selective that c=
an be
> > > > done?  Does it make sense to have THP backed stacks by default?  If=
 not,
> > > > who would be best at disabling?  A couple thoughts:
> > > > - The kernel could disable huge pages on stacks.  libpthread/glibc =
pass
> > > >   the unused flag MAP_STACK.  We could key off this and disable hug=
e pages.
> > > >   However, I'm sure there is somebody somewhere today that is getti=
ng better
> > > >   performance because they have huge pages backing their stacks.
> > > > - We could push this to glibc/libpthreads and have them use
> > > >   MADV_NOHUGEPAGE on thread stacks.  However, this also has the pot=
ential
> > > >   of regressing performance if somebody somewhere is getting better
> > > >   performance due to huge pages.
> > >
> > > Yes it seems it's always not safe to change a default behavior to me.
> > >
> > > For stack I really can't tell why it must be different here.  I assum=
e the
> > > problem is the wasted space and it exaggerates easily with N-threads.=
  But
> > > IIUC it'll be the same as thp to normal memories iiuc, e.g., there ca=
n be a
> > > per-thread mmap() of 2MB even if only 4K is used each, then if such m=
map()
> > > is populated by THP for each thread there'll also be a huge waste.
>
> I may be alone in my thinking here, but it seems that stacks by their nat=
ure
> are not generally good candidates for huge pages.  I am just thinking abo=
ut
> the 'normal' use case where stacks contain local function data and argume=
nts.
> Am I missing something, or are huge pages really a benefit here?
>
> Of course, I can imagine some thread with a large amount of frequently
> accessed data allocated on it's stack which could benefit from huge
> pages.  But, this seems to be an exception rather than the rule.
>
> I understand the argument that THP always means always and everywhere.
> It just seems that thread stacks may be 'special enough' to consider
> disabling by default

Just my drive-by 2c, but would agree with you here (at least wrt
hugepages not being good candidates, in general). A user mmap()'ing
memory has a lot more (direct) control over how they fault / utilize
the memory: you know when you're running out of space and can map more
space as needed. For these stacks, you're setting the stack size to
2MB just as a precaution so you can avoid overflow -- AFAIU there is
no intention of using the whole mapping (and looking at some data,
it's very likely you won't come close).

That said, why bother setting stack attribute to 2MiB in size if there
isn't some intention of possibly being THP-backed? Moreover, how did
it happen that the mappings were always hugepage-aligned here?

> --
> Mike Kravetz
>