Received-SPF: pass (google.com: domain of linux-kernel+bounces-34679-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Content-Type: text/plain;
	charset=us-ascii
Precedence: bulk
Mime-Version: 1.0
Subject: Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and
 initialization
From: Muchun Song <muchun.song@linux.dev>
In-Reply-To: <829fb129-f643-4960-a2da-cd38e5ee8f39@linux.dev>
Date: Tue, 23 Jan 2024 11:32:28 +0800
Cc: David Hildenbrand <david@redhat.com>,
 David Rientjes <rientjes@google.com>,
 Mike Kravetz <mike.kravetz@oracle.com>,
 Andrew Morton <akpm@linux-foundation.org>,
 Tim Chen <tim.c.chen@linux.intel.com>,
 Linux-MM <linux-mm@kvack.org>,
 LKML <linux-kernel@vger.kernel.org>,
 Gang Li <ligang.bdlg@bytedance.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <C79B8BB3-C1F8-4DFA-A084-C4B47486681F@linux.dev>
References: <20240118123911.88833-1-gang.li@linux.dev>
 <20240118123911.88833-7-gang.li@linux.dev>
 <ddf37da4-4cbc-478a-be9b-3060b0aebc90@linux.dev>
 <14e38e95-2bc6-4571-b502-4e3954b4bcc4@linux.dev>
 <849D7EA4-BCF4-4587-8A78-F3B35B63EAE9@linux.dev>
 <829fb129-f643-4960-a2da-cd38e5ee8f39@linux.dev>
To: Gang Li <gang.li@linux.dev>


> On Jan 23, 2024, at 10:12, Gang Li <gang.li@linux.dev> wrote:
>=20
> On 2024/1/22 19:30, Muchun Song wrote:
>>> On Jan 22, 2024, at 18:12, Gang Li <gang.li@linux.dev> wrote:
>>>=20
>>> On 2024/1/22 15:10, Muchun Song wrote:> On 2024/1/18 20:39, Gang Li =
wrote:
>>>>> +static void __init hugetlb_alloc_node(unsigned long start, =
unsigned long end, void *arg)
>>>>>   {
>>>>> -    unsigned long i;
>>>>> +    struct hstate *h =3D (struct hstate *)arg;
>>>>> +    int i, num =3D end - start;
>>>>> +    nodemask_t node_alloc_noretry;
>>>>> +    unsigned long flags;
>>>>> +    int next_node =3D 0;
>>>> This should be first_online_node which may be not zero.
>>>=20
>>> That's right. Thanks!
>>>=20
>>>>> -    for (i =3D 0; i < h->max_huge_pages; ++i) {
>>>>> -        if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
>>>>> +    /* Bit mask controlling how hard we retry per-node =
allocations.*/
>>>>> +    nodes_clear(node_alloc_noretry);
>>>>> +
>>>>> +    for (i =3D 0; i < num; ++i) {
>>>>> +        struct folio *folio =3D alloc_pool_huge_folio(h, =
&node_states[N_MEMORY],
>>>>> +                        &node_alloc_noretry, &next_node);
>>>>> +        if (!folio)
>>>>>               break;
>>>>> +        spin_lock_irqsave(&hugetlb_lock, flags);
>>>>> I suspect there will more contention on this lock when =
parallelizing.
>>>=20
>>> In the worst case, there are only 'numa node number' of threads in
>>> contention. And in my testing, it doesn't degrade performance, but
>>> rather improves performance due to the reduced granularity.
>> So, the performance does not change if you move the lock out of
>> loop?
>>=20
>=20
> If we move the lock out of loop, then multi-threading becomes =
single-threading, which definitely reduces performance.

No. I mean batching the pages into pool list just like =
prep_and_add_allocated_folios
does.

>=20
> ```
> +       spin_lock_irqsave(&hugetlb_lock, flags);
>        for (i =3D 0; i < num; ++i) {
>                struct folio *folio =3D alloc_pool_huge_folio(h, =
&node_states[N_MEMORY],
>                                                &node_alloc_noretry, =
&next_node);
>                if (!folio)
>                        break;
> -               spin_lock_irqsave(&hugetlb_lock, flags);
>                __prep_account_new_huge_page(h, folio_nid(folio));
>                enqueue_hugetlb_folio(h, folio);
> -               spin_unlock_irqrestore(&hugetlb_lock, flags);
>                cond_resched();
>        }
> +       spin_unlock_irqrestore(&hugetlb_lock, flags);
> }
> ```