Message-ID: <ff89f867-0709-d8bb-b6f5-51b2be4cc2dd@oracle.com>
Date:   Sun, 28 Nov 2021 20:31:35 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.0
Subject: Re: Commit 'hugetlbfs: extend the definition of hugepages parameter
 to support node allocation' breaks old numa less syntax of reserving
 hugepages on boot.
Content-Language: en-US
To:     Maxim Levitsky <mlevitsk@redhat.com>,
        Zhenguo Yao <yaozhenguo1@gmail.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org
References: <e52068a6e98031a386b5052a166a55c94fe940f6.camel@redhat.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
In-Reply-To: <e52068a6e98031a386b5052a166a55c94fe940f6.camel@redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cTZBd2NQSTZHMk5uV3o4cHNMT1JTTjFjeFhEU3I1dHZGbUdhMGl4OVg4WlZQ?=
 =?utf-8?B?ZWtESC9ubWxPQkpLQjZpSW1HWDl5M3BOZElrZXBMWmRDMVpBQVplaHdHRkZN?=
 =?utf-8?B?V3VkNUcrQlMxNE95R0huQklneG1RY0dKeWlpUnpLeUw2K2V0WG1EWmtSaDBS?=
 =?utf-8?B?Z2l5M1hyRGtxYlVuSjdoSisyRk9vOEdzc3l6ZnJ6M2lldGZlNS9WbVY0azdr?=
 =?utf-8?B?blV6d3BtZUVIbEtkYmVtV0VCeEJLVU50aW1BWDZPNkRKNVMrbDZCaTBVczlp?=
 =?utf-8?B?OEQ3YnU2dExXYXI5MGkxWnc3SkxBY3BzTDlWZVRTS3Q2aU5QS3pxQWY4NkV6?=
 =?utf-8?B?UDRsRlVqV3FRRjBXeVJIUWNZZkxlbGgrcG41d2dQQ3dCRFNUbjZWcHpwUnR3?=
 =?utf-8?B?VkxvOWdPM3JLVUM4d3E1ZVQ4bTloZlJRS0o4NmVpZTBDVERNTzBZbHBmTnhC?=
 =?utf-8?B?azd4dXFjdE8zUFovdnU2Y2xFSktoQ2xwN3JXcU1MKzFSMVFjendTYWVGY0dX?=
 =?utf-8?B?Q2NTZGtkU0o2UlRianFtTVZYcHQxWkVDYUluT2hXdkwrODF6THd5TXBFMUwr?=
 =?utf-8?B?MEpFeFRFM2p6dXhRVWJZQ2MvYlZXRUZsZXBzQTNKZjBYRjdjdWh2dTJXZ2ps?=
 =?utf-8?B?c2NmNzk0L1h0bWdLeGYxMmhSMTNxZlplbkF6ODZPWXVyK29UdFFlM0VIUGpV?=
 =?utf-8?B?Q2owejhFYVRpVU9DZ0FOdlk4OVUxcTVYeFIwRGYyL0NvaVFXU0tjNG1LSENS?=
 =?utf-8?B?bmpqdUVYa0NoaWMvQUtYQWtPZkx1d0VGTWNzTW1aQmNFaitaOFZxQXliYmRG?=
 =?utf-8?B?Zkdmb0I3QXNxZkZIWFdISWEwb05jVFhQbWJ4TUVnK29CdE8yRkdoUmYyTFRQ?=
 =?utf-8?B?MWdkeU5qOVhXbG1ZWW94cGdIUU40bS9EaDBVc2d2Q2xxQVZhS1lYb2ZBOVpn?=
 =?utf-8?B?UG5XNTJNYkdCU1BmYlZaaXBFRnFVQSttMGViYWcxeVNyemhIRFB5YkFnSCtZ?=
 =?utf-8?B?d2hGMkZUR3B6WjFtNWZjajdlTkxCRkdnc1J2RWR3dFJoQTQySG5RT3JzS3lz?=
 =?utf-8?B?K0dDdHVYRVBJUFVZZUh2TkF2SEhhZlpwRWh6U240L0ZTS0lwdWZlZ2F1bzla?=
 =?utf-8?B?MVY0THlRay9kcGlJWWx3cUltNlRYbmpXT2lpVVJjYytjMzRjMU8xZmdHdVJW?=
 =?utf-8?B?Zys3UEZNKzlnTTN6RWptNVpVaVd3Qm1odzJQRnZndFJGTXdZdTYwaUFoblNF?=
 =?utf-8?B?NXJ5K2xpdUZwNkFCWGdxbEk2RzFCc2xvazR0M2g3a3lrdlgxTzJHYUVDd1BM?=
 =?utf-8?B?dHZDcEhFL1Y3bWhzbjBJa09IVXJYZWViYTNLZWNJRGdMQVcrSElDU3dUM1h3?=
 =?utf-8?B?YlBreUkxZXFYc1ZnVVE2cHgwRWRMQ1ZlR3JLdGErUTY4YytRM3V3blJlVzhl?=
 =?utf-8?B?Z2NoVDhsQjE4UTN2bk5vZFhOcnRKYlZoSjRiNGlNUUhMajEyM3ZjdWRza2Zy?=
 =?utf-8?B?cE8ya1VEdGlEQ2JmcUE1NEhXL3hXTURTRDcrd1hoU204UWFNMUxNdWhjNjV2?=
 =?utf-8?B?dENWUGZTMXk5VWdUMkV4anpWUURjVUc0VUtVZjNYR3pJWUsyNWlmMUNkVjJC?=
 =?utf-8?B?N05UV2xybk9RWjgyNWxqZEhrc3BEQ1pNU254c1pEOWt5ZWR4cUtWMW4yc0Q3?=
 =?utf-8?B?dHlTOXJGZHd2ZFdMSXkrMnY0MkpCeDNrVWZIRmliK2YraU5NK0lseVVQcUkz?=
 =?utf-8?B?UEFmNFJ1RENHU3hObHZYTjJkbnFnbWlmSEhmVjBNRURlYmk2VlRieWZTdWtj?=
 =?utf-8?B?R3lldU50QUd6TVRRNTR3SUR0WEdJK0pXMDRnQ0Z3QTVMTkxPVHdPNWVqdXJm?=
 =?utf-8?B?ZW9WSWxFNmkvTU1LWHFjRi9oZ1VNYVhBZjZLSHk5cGhRdjJSZXZaYk9VMStJ?=
 =?utf-8?B?N3dqMThvckVoOWRXQ24raUZ2YytGMGR3RlYwZXc2RXFESlAyakg5SVc5THdv?=
 =?utf-8?B?b1V0anMvZy9IVUsraXc5N1FGSWFJS0VUd1EwY3BSUENxNDdTMXBmQVhGUFRG?=
 =?utf-8?B?dmpxUkhrOXU1TEVSVW1PNmZYakRwSWpOTFpKZUJCQllLUkQydEhyaDM1YTVP?=
 =?utf-8?B?MUsxdUNiTUc4dk45a1RtZWdxMnNRR1J6Q2dDcldpd1F0S3owaTdvM3lZb0pr?=
 =?utf-8?Q?3ODYPg+UAZF213aMCo9qpBQ=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1da73a50-f2a3-41a0-87fd-08d9b2f12211
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2021 04:31:37.9765
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KbsxJM2k0OpPMuO8vNi3zGAHgjczG0hxPkeS42gpWr88jUZ5j8Y/UCDgfHTLFbGB4dRs5tKyFLrhmwwzeyeTPA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3206
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10182 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 malwarescore=0
 spamscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111290019
X-Proofpoint-GUID: 27-kufWTHq7bWb-OtPiByrenepSj6eAi
X-Proofpoint-ORIG-GUID: 27-kufWTHq7bWb-OtPiByrenepSj6eAi
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/28/21 03:18, Maxim Levitsky wrote:
> 
> dmesg prints this:
> 
> HugeTLB: allocating 64 of page size 1.00 GiB failed.  Only allocated 0 hugepages
> 
> Huge pages were allocated on kernel command line (1/2 of 128GB system):
> 
> 'default_hugepagesz=1G hugepagesz=1G hugepages=64'
> 
> This is 3970X and no real support/need for NUMA, thus only fake NUMA node 0 is present.
> 
> Reverting the commit helps.
> 
> New syntax also works ( hugepages=0:64 )
> 
> I can test any patches for this bug.

Argh!  I think preallocation of gigantic pages on all systems with only
a single node is broken.  The issue is at the beginning of
__alloc_bootmem_huge_page:

int __alloc_bootmem_huge_page(struct hstate *h, int nid)
{
        struct huge_bootmem_page *m = NULL; /* initialize for clang */
        int nr_nodes, node;

        if (nid >= nr_online_nodes)
                return 0;

Without using the node specific syntax, nid == NUMA_NO_NODE == -1.  For the
comparison, nid will be converted to an unsigned into to match nr_online_nodes
so we will immediately return 0 instead of doing the allocations.

Zhenguo Yao,
Can you verify and perhaps put together a patch?does

> 
> Also unrelated, is there any progress on allocating 1GB pages on demand so that I could
> allocate them only when I run a VM?

That should be possible.  Such support was added back in 2014 with commit
944d9fec8d7a "hugetlb: add support for gigantic page allocation at runtime".

> 
> i don't mind having these pages to be marked as to be used for userspace only,
> since as far as I remember its the kernel usage that makes some page unmoveable.
> 

Of course, finding 1GB of contiguous space for a gigantic page is often
difficult at runtime.  So, allocations are likely to fail the longer the
system is up and running and fragmentation increases.

> Last time (many years ago) I tried to create a zone with only userspace pages
> (I don't remember what options I used) but it didn't work.

Not too long ago, support was added to use CMA for gigantic page allocation.
See commit cf11e85fc08c "mm: hugetlb: optionally allocate gigantic hugepages
using cma".  This sounds like something you might want to try.
-- 
Mike Kravetz

> 
> Is there a way to debug what is causing unmoveable pages and doesn't let
> /proc/sys/vm/nr_hugepages work (I tried it today and as usual the number
> it can allocate steadly decreases over time).