Received-SPF: pass (google.com: domain of linux-kernel+bounces-32931-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Content-Type: text/plain;
	charset=us-ascii
Precedence: bulk
Mime-Version: 1.0
Subject: Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and
 initialization
From: Muchun Song <muchun.song@linux.dev>
In-Reply-To: <14e38e95-2bc6-4571-b502-4e3954b4bcc4@linux.dev>
Date: Mon, 22 Jan 2024 19:30:39 +0800
Cc: David Hildenbrand <david@redhat.com>,
 David Rientjes <rientjes@google.com>,
 Mike Kravetz <mike.kravetz@oracle.com>,
 Andrew Morton <akpm@linux-foundation.org>,
 Tim Chen <tim.c.chen@linux.intel.com>,
 Linux-MM <linux-mm@kvack.org>,
 LKML <linux-kernel@vger.kernel.org>,
 ligang.bdlg@bytedance.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <849D7EA4-BCF4-4587-8A78-F3B35B63EAE9@linux.dev>
References: <20240118123911.88833-1-gang.li@linux.dev>
 <20240118123911.88833-7-gang.li@linux.dev>
 <ddf37da4-4cbc-478a-be9b-3060b0aebc90@linux.dev>
 <14e38e95-2bc6-4571-b502-4e3954b4bcc4@linux.dev>
To: Gang Li <gang.li@linux.dev>


> On Jan 22, 2024, at 18:12, Gang Li <gang.li@linux.dev> wrote:
>=20
> On 2024/1/22 15:10, Muchun Song wrote:> On 2024/1/18 20:39, Gang Li =
wrote:
>>> +static void __init hugetlb_alloc_node(unsigned long start, unsigned =
long end, void *arg)
>>>   {
>>> -    unsigned long i;
>>> +    struct hstate *h =3D (struct hstate *)arg;
>>> +    int i, num =3D end - start;
>>> +    nodemask_t node_alloc_noretry;
>>> +    unsigned long flags;
>>> +    int next_node =3D 0;
>> This should be first_online_node which may be not zero.
>=20
> That's right. Thanks!
>=20
>>> -    for (i =3D 0; i < h->max_huge_pages; ++i) {
>>> -        if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
>>> +    /* Bit mask controlling how hard we retry per-node =
allocations.*/
>>> +    nodes_clear(node_alloc_noretry);
>>> +
>>> +    for (i =3D 0; i < num; ++i) {
>>> +        struct folio *folio =3D alloc_pool_huge_folio(h, =
&node_states[N_MEMORY],
>>> +                        &node_alloc_noretry, &next_node);
>>> +        if (!folio)
>>>               break;
>>> +        spin_lock_irqsave(&hugetlb_lock, flags);
>> > I suspect there will more contention on this lock when =
parallelizing.
>=20
> In the worst case, there are only 'numa node number' of threads in
> contention. And in my testing, it doesn't degrade performance, but
> rather improves performance due to the reduced granularity.

So, the performance does not change if you move the lock out of
loop?

>=20
>> I want to know why you chose to drop prep_and_add_allocated_folios()
>> call in the original hugetlb_pages_alloc_boot()?
>=20
> Splitting him to parallelize hugetlb_vmemmap_optimize_folios.

Unfortunately, HVO should be enabled before pages go to the pool list.

>=20
>>> +static unsigned long __init hugetlb_pages_alloc_boot(struct hstate =
*h)
>>> +{
>>> +    struct padata_mt_job job =3D {
>>> +        .fn_arg        =3D h,
>>> +        .align        =3D 1,
>>> +        .numa_aware    =3D true
>>> +    };
>>> +
>>> +    job.thread_fn    =3D hugetlb_alloc_node;
>>> +    job.start    =3D 0;
>>> +    job.size    =3D h->max_huge_pages;
>>> +    job.min_chunk    =3D h->max_huge_pages / =
num_node_state(N_MEMORY) / 2;
>>> +    job.max_threads    =3D num_node_state(N_MEMORY) * 2;
>> I am curious the magic number of 2 used in assignments of ->min_chunk
>> and ->max_threads, does it from your experiment? I thinke it should
>> be a comment here.
>=20
> This is tested and I can perform more detailed tests and provide data.
>=20
>> And I am also sceptical about the optimization for a small amount of
>> allocation of hugepages. Given 4 hugepags needed to be allocated on =
UMA
>> system, job.min_chunk will be 2, job.max_threads will be 2. Then, 2
>> workers will be scheduled, however each worker will just allocate 2 =
pages,
>> how much the cost of scheduling? What if allocate 4 pages in single
>> worker? Do you have any numbers on parallelism vs non-parallelism in
>> a small allocation case? If we cannot gain from this case, I think we =
shold
>> assign a reasonable value to ->min_chunk based on experiment.
>> Thanks.
>>=20
>=20
> That's a good suggestion, I'll run some tests and choose the best
> values.
>=20
>=20