Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 14 Apr 2022 12:04:32 +0800
From:   Muchun Song <songmuchun@bytedance.com>
To:     Andrew Morton <akpm@linux-foundation.org>
Cc:     corbet@lwn.net, mike.kravetz@oracle.com, mcgrof@kernel.org,
        keescook@chromium.org, yzaikin@google.com, osalvador@suse.de,
        david@redhat.com, masahiroy@kernel.org, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        duanxiongchun@bytedance.com, smuchun@gmail.com
Subject: Re: [PATCH v8 4/4] mm: hugetlb_vmemmap: add hugetlb_optimize_vmemmap
 sysctl
Message-ID: <YledUEAa6cPBri52@FVFYT0MHHV2J.usts.net>
References: <20220413144748.84106-1-songmuchun@bytedance.com>
 <20220413144748.84106-5-songmuchun@bytedance.com>
 <20220413121051.a363193c726451115c634a69@linux-foundation.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220413121051.a363193c726451115c634a69@linux-foundation.org>
Precedence: bulk

On Wed, Apr 13, 2022 at 12:10:51PM -0700, Andrew Morton wrote:
> On Wed, 13 Apr 2022 22:47:48 +0800 Muchun Song <songmuchun@bytedance.com> wrote:
> 
> > We must add hugetlb_free_vmemmap=on (or "off") to the boot cmdline and
> > reboot the server to enable or disable the feature of optimizing vmemmap
> > pages associated with HugeTLB pages.  However, rebooting usually takes a
> > long time.  So add a sysctl to enable or disable the feature at runtime
> > without rebooting.
> 
> Do we really need this feature?  Really?  What's the use case and what
> is the end-user value?
>

We know that this feature is disabled by default without passing
"hugetlb_free_vmemmap=on" to the boot cmdline. When we (ByteDance)
deliver the servers to the users who want to enable this feature,
they have to configure the grub (change boot cmdline) and reboot the
servers, whereas rebooting usually takes a long time (we have
thousands of servers). It's a very bad experience for the users.
So we need a approach to enable this feature after rebooting. This
is a use case in our practical environment.

> Presumably CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP worsens things for some
> setups/workloads?  Please tell us much more about that.  What is the
> magnitude of the deoptimization?
>

Most workloads are allocated HugeTLB from the pool not the buddy
allocator, which are the users of this feature. However if the
use case is that HugeTLB pages are allocated 'on the fly' instead
of being pulled from the HugeTLB pool, those workloads would be
affected with this feature enabled.  Those workloads could be
identified by the characteristics of they never explicitly allocating
huge pages with 'nr_hugepages' but only set 'nr_overcommit_hugepages'
and then let the pages be allocated from the buddy allocator at
fault time.  I don't know what the specific workload is. But I
can confirm it is a real use case from the commit 099730d67417.
For those workloads, the page fault time could be ~2x slower than
before. I suspect those users want to disable this feature
if the system has enabled this before if they don't think the memory
savings benefit is enough to make up for the performance drop.

Do you think it's ok if I put those information in the commit
log or vm.rst?

Thanks.