Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Huang, Ying" <ying.huang@intel.com>
To:     Ryan Roberts <ryan.roberts@arm.com>
Cc:     David Hildenbrand <david@redhat.com>,
        Matthew Wilcox <willy@infradead.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Yin Fengwei <fengwei.yin@intel.com>,
        "Yu Zhao" <yuzhao@google.com>,
        Catalin Marinas <catalin.marinas@arm.com>,
        "Will Deacon" <will@kernel.org>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Yang Shi <shy828301@gmail.com>,
        <linux-arm-kernel@lists.infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Alexander Zhu <alexlzhu@fb.com>
Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
References: <20230703135330.1865927-1-ryan.roberts@arm.com>
        <20230703135330.1865927-5-ryan.roberts@arm.com>
        <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com>
        <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com>
        <ZKgZrNuxuq4ACvIb@casper.infradead.org>
        <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com>
        <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com>
Date:   Mon, 10 Jul 2023 11:03:01 +0800
In-Reply-To: <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> (Ryan Roberts's
        message of "Fri, 7 Jul 2023 16:13:52 +0100")
Message-ID: <87y1jofoyi.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Ryan Roberts <ryan.roberts@arm.com> writes:

> On 07/07/2023 15:07, David Hildenbrand wrote:
>> On 07.07.23 15:57, Matthew Wilcox wrote:
>>> On Fri, Jul 07, 2023 at 01:29:02PM +0200, David Hildenbrand wrote:
>>>> On 07.07.23 11:52, Ryan Roberts wrote:
>>>>> On 07/07/2023 09:01, Huang, Ying wrote:
>>>>>> Although we can use smaller page order for FLEXIBLE_THP, it's hard to
>>>>>> avoid internal fragmentation completely.=C2=A0 So, I think that fina=
lly we
>>>>>> will need to provide a mechanism for the users to opt out, e.g.,
>>>>>> something like "always madvise never" via
>>>>>> /sys/kernel/mm/transparent_hugepage/enabled.=C2=A0 I'm not sure whet=
her it's
>>>>>> a good idea to reuse the existing interface of THP.
>>>>>
>>>>> I wouldn't want to tie this to the existing interface, simply because=
 that
>>>>> implies that we would want to follow the "always" and "madvise" advic=
e too;
>>>>> That
>>>>> means that on a thp=3Dmadvise system (which is certainly the case for=
 android and
>>>>> other client systems) we would have to disable large anon folios for =
VMAs that
>>>>> haven't explicitly opted in. That breaks the intention that this shou=
ld be an
>>>>> invisible performance boost. I think it's important to set the policy=
 for
>>>>> use of
>>>>
>>>> It will never ever be a completely invisible performance boost, just l=
ike
>>>> ordinary THP.
>>>>
>>>> Using the exact same existing toggle is the right thing to do. If some=
one
>>>> specify "never" or "madvise", then do exactly that.
>>>>
>>>> It might make sense to have more modes or additional toggles, but
>>>> "madvise=3Dnever" means no memory waste.
>>>
>>> I hate the existing mechanisms.=C2=A0 They are an abdication of our
>>> responsibility, and an attempt to blame the user (be it the sysadmin
>>> or the programmer) of our code for using it wrongly.=C2=A0 We should not
>>> replicate this mistake.
>>=20
>> I don't agree regarding the programmer responsibility. In some cases the
>> programmer really doesn't want to get more memory populated than request=
ed --
>> and knows exactly why setting MADV_NOHUGEPAGE is the right thing to do.
>>=20
>> Regarding the madvise=3Dnever/madvise/always (sys admin decision), memor=
y waste
>> (and nailing down bugs or working around them in customer setups) have b=
een very
>> good reasons to let the admin have a word.
>>=20
>>>
>>> Our code should be auto-tuning.=C2=A0 I posted a long, detailed outline=
 here:
>>> https://lore.kernel.org/linux-mm/Y%2FU8bQd15aUO97vS@casper.infradead.or=
g/
>>>
>>=20
>> Well, "auto-tuning" also should be perfect for everybody, but once reali=
ty
>> strikes you know it isn't.
>>=20
>> If people don't feel like using THP, let them have a word. The "madvise"=
 config
>> option is probably more controversial. But the "always vs. never" absolu=
tely
>> makes sense to me.
>>=20
>>>> I remember I raised it already in the past, but you *absolutely* have =
to
>>>> respect the MADV_NOHUGEPAGE flag. There is user space out there (for
>>>> example, userfaultfd) that doesn't want the kernel to populate any
>>>> additional page tables. So if you have to respect that already, then a=
lso
>>>> respect MADV_HUGEPAGE, simple.
>>>
>>> Possibly having uffd enabled on a VMA should disable using large folios,
>>=20
>> There are cases where we enable uffd *after* already touching memory (po=
stcopy
>> live migration in QEMU being the famous example). That doesn't fly.
>>=20
>>> I can get behind that.=C2=A0 But the notion that userspace knows what i=
t's
>>> doing ... hahaha.=C2=A0 Just ignore the madvise flags.=C2=A0 Userspace =
doesn't
>>> know what it's doing.
>>=20
>> If user space sets MADV_NOHUGEPAGE, it exactly knows what it is doing ..=
. in
>> some cases. And these include cases I care about messing with sparse VM =
memory :)
>>=20
>> I have strong opinions against populating more than required when user s=
pace set
>> MADV_NOHUGEPAGE.
>
> I can see your point about honouring MADV_NOHUGEPAGE, so think that it is
> reasonable to fallback to allocating an order-0 page in a VMA that has it=
 set.
> The app has gone out of its way to explicitly set it, after all.
>
> I think the correct behaviour for the global thp controls (cmdline and sy=
sfs)
> are less obvious though. I could get on board with disabling large anon f=
olios
> globally when thp=3D"never". But for other situations, I would prefer to =
keep
> large anon folios enabled (treat "madvise" as "always"),

If we have some mechanism to auto-tune the large folios usage, for
example, detect the internal fragmentation and split the large folio,
then we can use thp=3D"always" as default configuration.  If my memory
were correct, this is what Johannes and Alexander is working on.

> with the argument that
> their order is much smaller than traditional THP and therefore the intern=
al
> fragmentation is significantly reduced.

Do you have any data for this?

> I really don't want to end up with user
> space ever having to opt-in (with MADV_HUGEPAGE) to see the benefits of l=
arge
> anon folios.
>
> I still feel that it would be better for the thp and large anon folio con=
trols
> to be independent though - what's the argument for tying them together?
>

Best Regards,
Huang, Ying