Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
MIME-Version: 1.0
References: <20230906112605.2286994-1-usama.arif@bytedance.com>
In-Reply-To: <20230906112605.2286994-1-usama.arif@bytedance.com>
From:   Pasha Tatashin <pasha.tatashin@soleen.com>
Date:   Fri, 22 Sep 2023 10:42:44 -0400
Message-ID: <CA+CK2bAfV=tpv_DK1621pU26Gy09zR+4E=AzR76Y9=zNYUc8SA@mail.gmail.com>
Subject: Re: [v4 0/4] mm: hugetlb: Skip initialization of gigantic tail struct
 pages if freed by HVO
To:     Usama Arif <usama.arif@bytedance.com>
Cc:     linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com,
        rppt@kernel.org, linux-kernel@vger.kernel.org,
        songmuchun@bytedance.com, fam.zheng@bytedance.com,
        liangma@liangbit.com, punit.agrawal@bytedance.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, Sep 6, 2023 at 7:26=E2=80=AFAM Usama Arif <usama.arif@bytedance.com=
> wrote:
>
> This series moves the boot time initialization of tail struct pages of a
> gigantic page to later on in the boot. Only the
> HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page) - 1 tail struct pages
> are initialized at the start. If HVO is successful, then no more tail str=
uct
> pages need to be initialized. For a 1G hugepage, this series avoid
> initialization of 262144 - 63 =3D 262081 struct pages per hugepage.
>
> When tested on a 512G system (which can allocate max 500 1G hugepages), t=
he
> kexec-boot time with HVO and DEFERRED_STRUCT_PAGE_INIT enabled without th=
is
> patchseries to running init is 3.9 seconds. With this patch it is 1.2 sec=
onds.
> This represents an approximately 70% reduction in boot time and will
> significantly reduce server downtime when using a large number of
> gigantic pages.

My use case is different, but this patch series benefits it. I have a
virtual machines with a large number of hugetlb pages. The RSS size of
the VM after boot is much smaller with this series:

Before: 9G
After: 600M

The VM has 500 1G pages, and 512G total RAM. I would add this to the
description, that this series can help reduce the VM overhead and boot
performance for those who are using hugetlb pages in the VMs.

Also, DEFERRED_STRUCT_PAGE_INIT is a requirement for this series to
work, and should be added into documentation.

Pasha

> Thanks,
> Usama
>
> [v3->v4]:
> - rebase ontop of patch "hugetlb: set hugetlb page flag before optimizing=
 vmemmap".
> - freeze head struct page ref count.
> - Change order of operations to initialize head struct page -> initialize
> the necessary tail struct pages -> attempt HVO -> initialize the rest of =
the
> tail struct pages if HVO fails.
> - (Mike Rapoport and Muchun Song) remove "_vmemmap" suffix from memblock =
reserve
> noinit flags anf functions.
>
> [v2->v3]:
> - (Muchun Song) skip prep of struct pages backing gigantic hugepages
> at boot time only.
> - (Muchun Song) move initialization of tail struct pages to after
> HVO is attempted.
>
> [v1->v2]:
> - (Mike Rapoport) Code quality improvements (function names, arguments,
> comments).
>
> [RFC->v1]:
> - (Mike Rapoport) Change from passing hugepage_size in
> memblock_alloc_try_nid_raw for skipping struct page initialization to
> using MEMBLOCK_RSRV_NOINIT flag
>
> Usama Arif (4):
>   mm: hugetlb_vmemmap: Use nid of the head page to reallocate it
>   memblock: pass memblock_type to memblock_setclr_flag
>   memblock: introduce MEMBLOCK_RSRV_NOINIT flag
>   mm: hugetlb: Skip initialization of gigantic tail struct pages if
>     freed by HVO
>
>  include/linux/memblock.h |  9 ++++++
>  mm/hugetlb.c             | 61 ++++++++++++++++++++++++++++++++++------
>  mm/hugetlb_vmemmap.c     |  4 +--
>  mm/hugetlb_vmemmap.h     |  9 +++---
>  mm/internal.h            |  3 ++
>  mm/memblock.c            | 48 ++++++++++++++++++++++---------
>  mm/mm_init.c             |  2 +-
>  7 files changed, 107 insertions(+), 29 deletions(-)
>
> --
> 2.25.1
>
>