Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH 2/5] mm: hugetlb: introduce helpers to preallocate page
 tables from bootmem allocator
To:     Muchun Song <songmuchun@bytedance.com>, akpm@linux-foundation.org,
        osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com,
        david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com,
        corbet@lwn.net
Cc:     duanxiongchun@bytedance.com, fam.zheng@bytedance.com,
        zhengqi.arch@bytedance.com, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org
References: <20210609121310.62229-1-songmuchun@bytedance.com>
 <20210609121310.62229-3-songmuchun@bytedance.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
Message-ID: <7c04420e-c564-7e19-6c2e-ea81dd51d396@oracle.com>
Date:   Thu, 10 Jun 2021 15:13:03 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.1.1
In-Reply-To: <20210609121310.62229-3-songmuchun@bytedance.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from [192.168.2.112] (50.38.35.18) by MWHPR1601CA0015.namprd16.prod.outlook.com (2603:10b6:300:da::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Thu, 10 Jun 2021 22:13:05 +0000
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: a8dfab54-0567-4049-48c3-08d92c5cec01
X-MS-TrafficTypeDiagnostic: BY5PR10MB3987:
X-Microsoft-Antispam-PRVS: <BY5PR10MB3987492354AE3E434D024FBFE2359@BY5PR10MB3987.namprd10.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:5797;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: yOt6n00kMeX6p/80t8bf39d/EnI928GV+6KpWARrPcz1qvPGHllIOJ16n9ILsWtrP5BxclDKzJDLRmjEBX+qFZguDBkZ5wnFkkRd2S/fQsYXJCmI+b1SH/ocNBoEcyWwMfwMp3K4OOInSo0qrTDzdy/taVVBANk5w9qHp+F6CPR/wpW2Jf2vle9xzrDVfGzWzEIgdPmUnJRwV1vxeUW/DRIoddU06l8zoUkdZ4YV2j+ReaiUDB14ZmczM9fdKYOLQixo5+UuvTEink+mKVj6r3U20+RcVkdohnxXPTcjJ8qBMUJorit1CzeG6lkpzm6yACgntTfJx2zfBns0nHqmvI8GUjseE1iBUdwVAhKUisYpegqbsfG51tvgXmp8NkjE8Vi5183tr+gKb6TbF9qz6b/4qsFH2picWFz58R8pNdm621b3tBJQDNtPknXd2ZK3st5+JA7jlpLAvzdlGqTHod8im3+mOFlcqr0JCwNGkJ7v2BKaFs2S1g+unbRqpYW983qoml9lxf+vLDtYvcvq2I69XdcxwVoCpCAqOdXlezN0E+9ntZOl8fGbT9Q/Ezd+JW3kHp8OeU5K3kf8pK13lfURzon45n6GS0i1mJgYu4GpRQePjHul9iUk/WFHDEWnNwybbrgIX8EJiEnf6ATIf+P6DzcxyXGoB9RzCVd9FxkP9UUSv4/prSqC+Dnq2EVPpwPwyZdBBiVfZDla38s9GCKNu2JGdwNF1ZgJmzIU7Jo=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(136003)(39860400002)(376002)(346002)(396003)(26005)(16526019)(44832011)(186003)(2906002)(52116002)(5660300002)(6486002)(38350700002)(2616005)(66476007)(66556008)(66946007)(31686004)(956004)(86362001)(8936002)(38100700002)(8676002)(478600001)(7416002)(31696002)(83380400001)(316002)(4326008)(53546011)(36756003)(16576012)(43740500002)(45980500001);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VlJnQldWUjFDdmZXbzZUR01wN2FWaDVxaXQrZnBkOTVhUWhndmRZSUpuMEJ4?=
 =?utf-8?B?aitRbzZwaXhqVFpUM3JtYnNZYXovSVNYV3VBdlZUWHd5N1RRb1BkZzV5K1Bh?=
 =?utf-8?B?cEx4eEg0a3BKNGVKYnJwUnJUblh2bVArZW0waENEV0M2VDF1dDl6NGhUWWRH?=
 =?utf-8?B?bGZzbFN3Y05XL1haeHo0UGl5OGlqRzB2aEZBWVplNkhLL3lla1IxVDg2MGh6?=
 =?utf-8?B?ei95dkQ4SFlyYXU2RXVwUDQ1TDRqaHlQQzFBVUMzc1hvRktaQm5JZTBXZWQ1?=
 =?utf-8?B?ZDl5V1pWaUx2NGlHVi84ODJPUUpEVkVYaW5OL2lmTTZMQ09aWWZpbjNuMWJ0?=
 =?utf-8?B?ajBzbTVaWHo4OURmZFJUWFlpUm1XLzdUSkJsL1hxcE1FNi9KMi9YdG0rWVZJ?=
 =?utf-8?B?cEJISlVZWEk0akYwUUV2M2JYSWR6WStNLzVLeXQ5cEdLMFcramNOaXVoaFRl?=
 =?utf-8?B?YkJYRHdvaDBFTW9ycTZ3K3puL0JUbTg4bU1XL1RCVGhjSE9pMHY0azhaVC9J?=
 =?utf-8?B?WkU5OWZRV2U1bmF4Zi9lK2NPeklESVg1eHplTHpmZmxEeHY4Slk2dUhQNmsr?=
 =?utf-8?B?WmJvM2dNSlU1VWtKN3FNRUJQMkJFc1htUFBQSWtORFR2R0xCdzlnb0d0VlJM?=
 =?utf-8?B?aEtHWElpTUZSdzVKRWQwR08yRVp1RHFuVm42c3hHeG9JYzhzNjluQjJ5dkEy?=
 =?utf-8?B?dTUwTlVmR3VFdmhSbXhxNkVoOVRBdDVOaFgyOWpINzFlOVNiZGVHSnp0VmhS?=
 =?utf-8?B?MW9HT0Z5V2JDUnFHSlBMU3VaV0p5QXk4NmtFMG1uczdwTzg2S0hORmlKS0d2?=
 =?utf-8?B?MzlJU0N6aGQ2QkZtWURWMWU1TG56NHV6TU1VSVFIVGdNcFpRVHl3OUJ4NExs?=
 =?utf-8?B?YzBQUVNwd3BDeWpTblVnRGo2ampOTk0vUmJlYkxZcXBMM1J3UTBhSThYYzFO?=
 =?utf-8?B?bUV4VHVJTEhiQTdGK3l2dm5oMWhlUHZKM0ZpUkFxOVdjbTdoL0gyc3FuL1Av?=
 =?utf-8?B?SlJYOERiM2c3VzNJT3hQNEZjcGFRSHRJTnhHTUtiR0I3VENObHAzeE5RTHE0?=
 =?utf-8?B?MUJxVTl1VXRodUhqbGhueTJvWHNCVVE1NnRQUmVxbzc2SVV4UlRlczhRVWJY?=
 =?utf-8?B?emJSdnc2Z3JFcnpzYUVNUWYrcE53dHdESUNhTU45N1VkM2hSSHNyVTdMTnJK?=
 =?utf-8?B?bWpjNVNKMXNwTVNEZy8xSkpqYUlGM3ozelJPWWNqTTdWRDBOUFdMNWdmSEdM?=
 =?utf-8?B?ZXpqUTIwVTF2eWF6Znk4bVhhMGkvZVVONkRiczgwV1VEanEzSEtmSllsU3ow?=
 =?utf-8?B?VFRLWWppN2VHbHUyREl4NHFmbldoWDZRNnlHWnpxWUpsNHpyVERLS0RZbzZB?=
 =?utf-8?B?QjEvLzVYeFpsbFlmeVRqSDdwb0VoVGJJcHoyNE1oOGtnNEtWcEoyTThnd2Uv?=
 =?utf-8?B?ZmQ1My9FTGpZQXQ5SGxUQ0ZJbHRFaWFvb25sNnVmSnlwUEFEaXBTdXQ5Rk8z?=
 =?utf-8?B?VVMwUThZK1RWWFlWTDl1UVREdGw5b3pybkZoNDB2QjRLTWY5QnNnbzJZM3Jl?=
 =?utf-8?B?Y3pFVXlTZy9UbFV2OWtOalE5cGFtcUFiUlRTYndGUDhpRFBFQVRVdjlmNXN1?=
 =?utf-8?B?WWt2dExYK1FmYy95YW9ZamVidXBYVFhZZmV0aVVyTUN6S2MvZDBnS0gvVzhs?=
 =?utf-8?B?VXUzVTc3Tk5WenkydmJIdkJzU3ZwWG5sa2J1dGJNZ1BvOWFsS08rYkFBQmk1?=
 =?utf-8?Q?z+/vT07JBs65GG9V1zLXAQYAHVwDLD8YNQgxq5O?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a8dfab54-0567-4049-48c3-08d92c5cec01
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2021 22:13:05.9319
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WU3W2mDoWYIIXR0JCG4+rS0B8ZoJvnNO8A0irbmPYEOTPANVLAma+KrWRv5LYzRlEoL+oTmfvfEror96hX8ABg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10011 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0
 adultscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2106100133
X-Proofpoint-GUID: kWsMcaPZ6butNh4lwKSve0oLJdohevuz
X-Proofpoint-ORIG-GUID: kWsMcaPZ6butNh4lwKSve0oLJdohevuz
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10011 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 clxscore=1015
 bulkscore=0 spamscore=0 mlxscore=0 adultscore=0 malwarescore=0
 phishscore=0 suspectscore=0 lowpriorityscore=0 mlxlogscore=999
 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2104190000 definitions=main-2106100132
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 6/9/21 5:13 AM, Muchun Song wrote:
> If we want to split the huge PMD of vmemmap pages associated with each
> gigantic page allocated from bootmem allocator, we should pre-allocate
> the page tables from bootmem allocator.

Just curious why this is necessary and a good idea?  Why not wait until
the gigantic pages allocated from bootmem are added to the pool to
allocate any necessary vmemmmap pages?

> the page tables from bootmem allocator. In this patch, we introduce
> some helpers to preallocate page tables for gigantic pages.
> 
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> ---
>  include/linux/hugetlb.h |  3 +++
>  mm/hugetlb_vmemmap.c    | 63 +++++++++++++++++++++++++++++++++++++++++++++++++
>  mm/hugetlb_vmemmap.h    | 13 ++++++++++
>  3 files changed, 79 insertions(+)
> 
> diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
> index 03ca83db0a3e..c27a299c4211 100644
> --- a/include/linux/hugetlb.h
> +++ b/include/linux/hugetlb.h
> @@ -622,6 +622,9 @@ struct hstate {
>  struct huge_bootmem_page {
>  	struct list_head list;
>  	struct hstate *hstate;
> +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP
> +	pte_t *vmemmap_pte;
> +#endif
>  };
>  
>  int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list);
> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
> index 628e2752714f..6f3a47b4ebd3 100644
> --- a/mm/hugetlb_vmemmap.c
> +++ b/mm/hugetlb_vmemmap.c
> @@ -171,6 +171,7 @@
>  #define pr_fmt(fmt)	"HugeTLB: " fmt
>  
>  #include <linux/list.h>
> +#include <linux/memblock.h>
>  #include <asm/pgalloc.h>
>  
>  #include "hugetlb_vmemmap.h"
> @@ -263,6 +264,68 @@ int vmemmap_pgtable_prealloc(struct hstate *h, struct list_head *pgtables)
>  	return -ENOMEM;
>  }
>  
> +unsigned long __init gigantic_vmemmap_pgtable_prealloc(void)
> +{
> +	struct huge_bootmem_page *m, *tmp;
> +	unsigned long nr_free = 0;
> +
> +	list_for_each_entry_safe(m, tmp, &huge_boot_pages, list) {
> +		struct hstate *h = m->hstate;
> +		unsigned int nr = pgtable_pages_to_prealloc_per_hpage(h);
> +		unsigned long size;
> +
> +		if (!nr)
> +			continue;
> +
> +		size = nr << PAGE_SHIFT;
> +		m->vmemmap_pte = memblock_alloc_try_nid(size, PAGE_SIZE, 0,
> +							MEMBLOCK_ALLOC_ACCESSIBLE,
> +							NUMA_NO_NODE);
> +		if (!m->vmemmap_pte) {
> +			nr_free++;
> +			list_del(&m->list);
> +			memblock_free_early(__pa(m), huge_page_size(h));

If we can not allocate the vmmmemap pages to split the PMD, then we will
not add the huge page to the pool.  Correct?

Perhaps I am thinking about this incorrectly, but this seems wrong.  We
already have everything we need to add the page to the pool.  vmemmap
reduction is an optimization.  So, the allocation failure is associated
with an optimization.  In this case, it seems like we should just skip
the optimization (vmemmap reduction) and proceed to add the page to the
pool?  It seems we do the same thing in subsequent patches.

Again, I could be thinking about this incorrectly.
-- 
Mike Kravetz