Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
MIME-Version: 1.0
References: <20231123065708.91345-1-luxu.kernel@bytedance.com> <94c2e04c-4c62-4ee1-8ae7-cbd675c5064e@app.fastmail.com>
In-Reply-To: <94c2e04c-4c62-4ee1-8ae7-cbd675c5064e@app.fastmail.com>
From:   Xu Lu <luxu.kernel@bytedance.com>
Date:   Mon, 27 Nov 2023 16:14:03 +0800
Message-ID: <CAPYmKFtA7XuDR3vzd1aw_RC1YY3hDgo-aw-DHw368mdDUtyW6A@mail.gmail.com>
Subject: Re: [External] Re: [RFC PATCH V1 00/11] riscv: Introduce 64K base page
To:     Arnd Bergmann <arnd@arndb.de>
Cc:     Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        Ard Biesheuvel <ardb@kernel.org>,
        Anup Patel <anup@brainfault.org>,
        Atish Patra <atishp@atishpatra.org>,
        dengliang.1214@bytedance.com, Xie Yongji <xieyongji@bytedance.com>,
        lihangjing@bytedance.com, Muchun Song <songmuchun@bytedance.com>,
        punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org,
        linux-riscv@lists.infradead.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Thanks a lot for your reply! And sorry for replying so late.

On Thu, Nov 23, 2023 at 5:30=E2=80=AFPM Arnd Bergmann <arnd@arndb.de> wrote=
:
>
> On Thu, Nov 23, 2023, at 07:56, Xu Lu wrote:
> > Some existing architectures like ARM supports base page larger than 4K
> > as their MMU supports more page sizes. Thus, besides hugetlb page and
> > transparent huge page, there is another way for these architectures to
> > enjoy the benefits of fewer TLB misses without worrying about cost of
> > splitting and merging huge pages. However, on architectures with only
> > 4K MMU, larger base page is unavailable now.
> >
> > This patch series attempts to break through the limitation of MMU and
> > supports larger base page on RISC-V, which only supports 4K page size
> > now.
> >
> > The key idea to implement larger base page based on 4K MMU is to
> > decouple the MMU page from the base page in view of kernel mm, which we
> > denote as software page. In contrary to software page, we denote the MM=
U
> > page as hardware page. Below is the difference between these two kinds
> > of pages.
>
> We have played with this on arm32, but the conclusion is that it's
> almost never worth the memory overhead, as most workloads end up
> using several times the amount of physical RAM after each small
> file in the page cache and any sparse populated anonymous memory
> area explodes to up to 16 times the size.
>
> On ppc64, using 64KB pages was way to get around limitations in
> their hashed MMU design, which had a much bigger performance impact
> because any page table access ends up being a cache miss. On arm64,
> there are some CPUs like the Fujitsu A64FX that are really bad at
> 4KB pages and don't support 16KB pages, so this is the only real
> option.
>
> You will see a notable performance benefit in synthetic benchmarks
> like speccpu with 64KB pages, or on specific computational
> workloads that have large densely packed memory chunks, but for
> real workloads, the usual answer is to just use transparent
> hugepages for larger mappings and a page size of no more than
> 16KB for the page cache.

Actually we did find actual performance benefits brought by 64K page
size in real business scenarios.
On the Ampere ARM server, when applying 64K base page size, we saw an
improvement of 2.5x for both qps and latency on redis, a performance
improvement of 10~20% on our own newsql database and 50% on object
storage.
For mysql, the qps increases about 14%, 17.5% and 20% for read-only,
write-only and random read/write workloads respectively. And the
latency reduces about 13.7%, 15.8% and 14.5% on average.
This is also why we chose to implement a similar feature on RISC-V in
the beginning.

>
> With the work going into using folios in the kernel (see e.g.
> https://lwn.net/Articles/932386/), even the workloads that
> benefit from 64KB base pages should be better off with 4KB
> pages and just using the TLB hints for large folios.

Maybe 64K page size combined with large folios can achieve more benefits.
As is mentioned in this patch[1], a 64K page size kernel combined with
large folios and THPs via cont pte can achieve speedup of 10.5x on
some memory-intensive workloads on arm64 SBSA server.

[1] https://lore.kernel.org/all/c507308d-bdd4-5f9e-d4ff-e96e4520be85@nvidia=
.com/

>
>      Arnd