Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [RFC PATCH 00/16] 1GB THP support on x86_64
To:     Roman Gushchin <guro@fb.com>, Michal Hocko <mhocko@suse.com>
Cc:     Zi Yan <ziy@nvidia.com>, linux-mm@kvack.org,
        Rik van Riel <riel@surriel.com>,
        "Kirill A.Shutemov" <kirill.shutemov@linux.intel.com>,
        Matthew Wilcox <willy@infradead.org>,
        Shakeel Butt <shakeelb@google.com>,
        Yang Shi <yang.shi@linux.alibaba.com>,
        David Nellans <dnellans@nvidia.com>,
        linux-kernel@vger.kernel.org
References: <20200902180628.4052244-1-zi.yan@sent.com>
 <20200903073254.GP4617@dhcp22.suse.cz>
 <20200903162527.GF60440@carbon.dhcp.thefacebook.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
Message-ID: <d64a67fd-17e0-644d-7b2b-2a557f0e35f7@oracle.com>
Date:   Thu, 3 Sep 2020 13:57:54 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.11.0
MIME-Version: 1.0
In-Reply-To: <20200903162527.GF60440@carbon.dhcp.thefacebook.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 9/3/20 9:25 AM, Roman Gushchin wrote:
> On Thu, Sep 03, 2020 at 09:32:54AM +0200, Michal Hocko wrote:
>> On Wed 02-09-20 14:06:12, Zi Yan wrote:
>>> From: Zi Yan <ziy@nvidia.com>
>>>
>>> Hi all,
>>>
>>> This patchset adds support for 1GB THP on x86_64. It is on top of
>>> v5.9-rc2-mmots-2020-08-25-21-13.
>>>
>>> 1GB THP is more flexible for reducing translation overhead and increasing the
>>> performance of applications with large memory footprint without application
>>> changes compared to hugetlb.
>>
>> Please be more specific about usecases. This better have some strong
>> ones because THP code is complex enough already to add on top solely
>> based on a generic TLB pressure easing.
> 
> Hello, Michal!
> 
> We at Facebook are using 1 GB hugetlbfs pages and are getting noticeable
> performance wins on some workloads.
> 
> Historically we allocated gigantic pages at the boot time, but recently moved
> to cma-based dynamic approach. Still, hugetlbfs interface requires more management
> than we would like to do. 1 GB THP seems to be a better alternative. So I definitely
> see it as a very useful feature.
> 
> Given the cost of an allocation, I'm slightly skeptical about an automatic
> heuristics-based approach, but if an application can explicitly mark target areas
> with madvise(), I don't see why it wouldn't work.
> 
> In our case we'd like to have a reliable way to get 1 GB THPs at some point
> (usually at the start of an application), and transparently destroy them on
> the application exit.

Hi Roman,

In your current use case at Facebook, are you adding 1G hugetlb pages to
the hugetlb pool and then using them within applications?  Or, are you
dynamically allocating them at fault time (hugetlb overcommit/surplus)?

Latency time for use of such pages includes:
- Putting together 1G contiguous
- Clearing 1G memory

In the 'allocation at fault time' mode you incur both costs at fault time.
If using pages from the pool, your only cost at fault time is clearing the
page.
-- 
Mike Kravetz