Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220805184016.2926168-1-alexlzhu@fb.com> <Yu1mcD6Jp4fCVEMi@casper.infradead.org>
 <0b16dbac6444bfcdfbeb4df4280354839bfe1a8f.camel@fb.com> <Yu1uabedm+NYnnAj@casper.infradead.org>
 <CA8C72B6-E509-4FB0-BEAA-C4368EB7A419@fb.com> <CAHbLzkp27aP4JYLPLzv2vtyzVe63bkhuZhw1jnxTF1Buvt4rew@mail.gmail.com>
 <fc108f58a4616d5d7d092a7c1f150069a92ee40c.camel@fb.com> <CAHbLzkqpn2ExBJuPD8sYJrEDCUU9=FE3GFh8kL3Bmax0KytKPw@mail.gmail.com>
 <1F8B9D85-A735-4832-AD58-CA4BD474248D@fb.com> <CAOUHufbGcuvJ8EfBkWrdCgeMCiceP0SxHZzcJnOt1Mw0P2w9kw@mail.gmail.com>
 <CAHbLzkqGq6LDDSCcFTJW9_O8dfAMje8_W6ddpDP9ESDQWUb07w@mail.gmail.com>
 <F28E23D0-C635-4A49-91EE-100AAA8C9EAD@fb.com> <CAOUHufYeuMN9As58BVwMKSN6viOZKReXNeCBgGeeL6ToWGsEKw@mail.gmail.com>
 <868F0874-70E8-4416-B39B-DA74C9D76A40@fb.com> <CAOUHufas1o1Ez9EDSjjxsO-w6Ph6Km8R8yOX+1_c=4LvG6DEWw@mail.gmail.com>
 <DD679B3A-BDF7-4EBD-AAC2-A663057AC8E3@fb.com> <CAOUHufbD-9PpQ+kuD=-8z-ptsrprjyThpkFe+4_NtFnzAjDG9g@mail.gmail.com>
 <DEB2F4F2-7F62-48F0-914D-5F71BFDBCBEE@fb.com> <3195C304-2140-4E5D-890D-AC55653193E5@fb.com>
 <CAOUHufaggjiYAywB646N5Rj48eSJDjir8oLwDVu039ymbLm0PA@mail.gmail.com>
In-Reply-To: <CAOUHufaggjiYAywB646N5Rj48eSJDjir8oLwDVu039ymbLm0PA@mail.gmail.com>
From:   Yang Shi <shy828301@gmail.com>
Date:   Thu, 11 Aug 2022 15:12:06 -0700
Message-ID: <CAHbLzkpRVOOGK2d0L25A4PKfEbHngm-WEWWH34UYVg1O40Tiig@mail.gmail.com>
Subject: Re: [PATCH v3] mm: add thp_utilization metrics to /proc/thp_utilization
To:     Yu Zhao <yuzhao@google.com>
Cc:     "Alex Zhu (Kernel)" <alexlzhu@fb.com>, Rik van Riel <riel@fb.com>,
        Kernel Team <Kernel-team@fb.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "willy@infradead.org" <willy@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        Ning Zhang <ningzhang@linux.alibaba.com>,
        Miaohe Lin <linmiaohe@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, Aug 11, 2022 at 2:55 PM Yu Zhao <yuzhao@google.com> wrote:
>
> On Thu, Aug 11, 2022 at 1:20 PM Alex Zhu (Kernel) <alexlzhu@fb.com> wrote=
:
> >
> > Hi Yu,
> >
> > I=E2=80=99ve updated your patch set from last year to work with folio a=
nd am testing it now. The functionality in split_huge_page() is the same as=
 what I have. Was there any follow up work done later?
>
> Yes, but it won't change the landscape any time soon (see below). So
> please feel free to continue along your current direction.
>
> > If not, I would like to incorporate this into what I have, and then res=
ubmit. Will reference the original patchset. We need this functionality for=
 the shrinker, but even the changes to split_huge_page() by itself it shoul=
d show some performance improvement when used by the existing deferred_spli=
t_huge_page().
>
> SGTM. Thanks!
>
> A side note:
>
> I'm working on a new mode: THP=3Dauto, meaning the kernel will detect
> internal fragmentation of 2MB compound pages to decide whether to map
> them by PMDs or split them under memory pressure. The general workflow
> of this new mode is as follows.

I tend to agree that avoiding allocating THP in the first place is the
preferred way to avoid internal fragmentation. But I got some
questions about your design/implementation:

>
> In the page fault path:
> 1. Compound pages are allocated as usual.
> 2. Each is mapped by 512 consecutive PTEs rather than a PMD.
> 3. There will be more TLB misses but the same number of page faults.
> 4. TLB coalescing can mitigate the performance degradation.

Why not just allocate base pages in the first place? Khugepaged has
max_pte_none tunable to detect internal fragmentation. If you worry
about zero page, you could add max_pte_zero tunable.

Or did you investigate whether the new MADV_COLLAPSE may be helpful or
not? It leaves the decision to the userspace.

>
> In khugepaged:
> 1. Check the dirty bit in the PTEs mapping a compound page, to
> determine its utilization.
> 2. Remap compound pages that meet a certain utilization threshold by
> PMDs in place, i.e., no migrations.
>
> In the reclaim path, e.g., MGLRU page table scanning:
> 1. Decide whether compound pages mapped by PTEs should be split based
> on their utilizations and memory pressure, e.g., reclaim priority.
> 2. Clean subpages should be freed directly after split, rather than swapp=
ed out.
>
> N.B.
> 1. This workflow relies on the dirty bit rather examining the content of =
a page.
> 2. Sampling can be done by periodically switching between a PMD and
> 512 consecutive PTEs.
> 3. It only needs to hold mmap_lock for read because this special mode
> (512 consecutive PTEs) is not considered the split mode.
> 4. Don't hold your breath :)
>
> Other references:
> 1. https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf
> 2. https://www.usenix.org/system/files/osdi21-hunter.pdf