Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <20220331065640.5777-1-songmuchun@bytedance.com>
 <20220331065640.5777-2-songmuchun@bytedance.com> <dd980687-a1ef-f4b5-bb29-da14f89c087e@arm.com>
In-Reply-To: <dd980687-a1ef-f4b5-bb29-da14f89c087e@arm.com>
From:   Muchun Song <songmuchun@bytedance.com>
Date:   Mon, 4 Apr 2022 20:01:20 +0800
Message-ID: <CAMZfGtX5NXudeAsYKU8mDtPn0+bw_3FhCsAW0PMBTqTTkp-fQQ@mail.gmail.com>
Subject: Re: [PATCH v4 2/2] arm64: mm: hugetlb: Enable HUGETLB_PAGE_FREE_VMEMMAP
 for arm64
To:     Anshuman Khandual <anshuman.khandual@arm.com>
Cc:     Will Deacon <will@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        David Hildenbrand <david@redhat.com>,
        "Bodeddula, Balasubramaniam" <bodeddub@amazon.com>,
        Oscar Salvador <osalvador@suse.de>,
        Mike Kravetz <mike.kravetz@oracle.com>,
        David Rientjes <rientjes@google.com>,
        Mark Rutland <mark.rutland@arm.com>,
        Catalin Marinas <catalin.marinas@arm.com>, james.morse@arm.com,
        Barry Song <21cnbao@gmail.com>,
        LAK <linux-arm-kernel@lists.infradead.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux Memory Management List <linux-mm@kvack.org>,
        Xiongchun duan <duanxiongchun@bytedance.com>,
        Fam Zheng <fam.zheng@bytedance.com>,
        Muchun Song <smuchun@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Apr 4, 2022 at 5:25 PM Anshuman Khandual
<anshuman.khandual@arm.com> wrote:
>
> Hello Muchun,
>
> On 3/31/22 12:26, Muchun Song wrote:
> > The feature of minimizing overhead of struct page associated with each
> > HugeTLB page aims to free its vmemmap pages (used as struct page) to
> > save memory, where is ~14GB/16GB per 1TB HugeTLB pages (2MB/1GB type).
>
> Enabling this feature saves us around 1.4/1.6 % memory but looking from
> other way around, unavailability of vmemmap backing pages (~1.4GB) when
> freeing up a corresponding HugeTLB page, could prevent ~1TB memory from
> being used as normal page form (requiring their own struct pages), thus
> forcing the HugeTLB page to remain as such ? Is not this problematic ?
>
> These additional 1TB memory in normal pages, from a HugeTLB dissolution
> could have eased the system's memory pressure without this feature being
> enabled.

You are right. If the system is already under heavy memory pressure, it could
prevent the user from freeing HugeTLB pages to the buddy allocator. If the
HugeTLB page are allocated from non-movable zone, this scenario may be
not problematic since once a HugeTLB page is freed, then the system will
have memory to be allocated to be used as vmemmap pages, subsequent
freeing of HugeTLB pages may be getting easier.  However, if the HUgeTLB
pages are allocated from the movable zone, then the thing becomes terrible,
which is documented in Documentation/admin-guide/mm/memory-hotplug.rst.

So there is a cmdline "hugetlb_free_vmemmap" to control if enabling this
feature.  The user should enable/disable this depending on their workload.

Thanks.