Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220805184016.2926168-1-alexlzhu@fb.com> <Yu1mcD6Jp4fCVEMi@casper.infradead.org>
 <0b16dbac6444bfcdfbeb4df4280354839bfe1a8f.camel@fb.com> <Yu1uabedm+NYnnAj@casper.infradead.org>
 <CA8C72B6-E509-4FB0-BEAA-C4368EB7A419@fb.com> <CAHbLzkp27aP4JYLPLzv2vtyzVe63bkhuZhw1jnxTF1Buvt4rew@mail.gmail.com>
 <fc108f58a4616d5d7d092a7c1f150069a92ee40c.camel@fb.com> <CAHbLzkqpn2ExBJuPD8sYJrEDCUU9=FE3GFh8kL3Bmax0KytKPw@mail.gmail.com>
 <1F8B9D85-A735-4832-AD58-CA4BD474248D@fb.com> <CAOUHufbGcuvJ8EfBkWrdCgeMCiceP0SxHZzcJnOt1Mw0P2w9kw@mail.gmail.com>
 <CAHbLzkqGq6LDDSCcFTJW9_O8dfAMje8_W6ddpDP9ESDQWUb07w@mail.gmail.com>
 <F28E23D0-C635-4A49-91EE-100AAA8C9EAD@fb.com> <CAOUHufYeuMN9As58BVwMKSN6viOZKReXNeCBgGeeL6ToWGsEKw@mail.gmail.com>
 <868F0874-70E8-4416-B39B-DA74C9D76A40@fb.com> <CAOUHufas1o1Ez9EDSjjxsO-w6Ph6Km8R8yOX+1_c=4LvG6DEWw@mail.gmail.com>
 <DD679B3A-BDF7-4EBD-AAC2-A663057AC8E3@fb.com> <CAOUHufbD-9PpQ+kuD=-8z-ptsrprjyThpkFe+4_NtFnzAjDG9g@mail.gmail.com>
 <DEB2F4F2-7F62-48F0-914D-5F71BFDBCBEE@fb.com> <3195C304-2140-4E5D-890D-AC55653193E5@fb.com>
In-Reply-To: <3195C304-2140-4E5D-890D-AC55653193E5@fb.com>
From:   Yu Zhao <yuzhao@google.com>
Date:   Thu, 11 Aug 2022 15:55:07 -0600
Message-ID: <CAOUHufaggjiYAywB646N5Rj48eSJDjir8oLwDVu039ymbLm0PA@mail.gmail.com>
Subject: Re: [PATCH v3] mm: add thp_utilization metrics to /proc/thp_utilization
To:     "Alex Zhu (Kernel)" <alexlzhu@fb.com>
Cc:     Yang Shi <shy828301@gmail.com>, Rik van Riel <riel@fb.com>,
        Kernel Team <Kernel-team@fb.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "willy@infradead.org" <willy@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        Ning Zhang <ningzhang@linux.alibaba.com>,
        Miaohe Lin <linmiaohe@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, Aug 11, 2022 at 1:20 PM Alex Zhu (Kernel) <alexlzhu@fb.com> wrote:
>
> Hi Yu,
>
> I=E2=80=99ve updated your patch set from last year to work with folio and=
 am testing it now. The functionality in split_huge_page() is the same as w=
hat I have. Was there any follow up work done later?

Yes, but it won't change the landscape any time soon (see below). So
please feel free to continue along your current direction.

> If not, I would like to incorporate this into what I have, and then resub=
mit. Will reference the original patchset. We need this functionality for t=
he shrinker, but even the changes to split_huge_page() by itself it should =
show some performance improvement when used by the existing deferred_split_=
huge_page().

SGTM. Thanks!

A side note:

I'm working on a new mode: THP=3Dauto, meaning the kernel will detect
internal fragmentation of 2MB compound pages to decide whether to map
them by PMDs or split them under memory pressure. The general workflow
of this new mode is as follows.

In the page fault path:
1. Compound pages are allocated as usual.
2. Each is mapped by 512 consecutive PTEs rather than a PMD.
3. There will be more TLB misses but the same number of page faults.
4. TLB coalescing can mitigate the performance degradation.

In khugepaged:
1. Check the dirty bit in the PTEs mapping a compound page, to
determine its utilization.
2. Remap compound pages that meet a certain utilization threshold by
PMDs in place, i.e., no migrations.

In the reclaim path, e.g., MGLRU page table scanning:
1. Decide whether compound pages mapped by PTEs should be split based
on their utilizations and memory pressure, e.g., reclaim priority.
2. Clean subpages should be freed directly after split, rather than swapped=
 out.

N.B.
1. This workflow relies on the dirty bit rather examining the content of a =
page.
2. Sampling can be done by periodically switching between a PMD and
512 consecutive PTEs.
3. It only needs to hold mmap_lock for read because this special mode
(512 consecutive PTEs) is not considered the split mode.
4. Don't hold your breath :)

Other references:
1. https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf
2. https://www.usenix.org/system/files/osdi21-hunter.pdf