Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
Date:   Tue, 12 Dec 2023 12:06:53 -0800
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     Gang Li <gang.li@linux.dev>
Cc:     David Hildenbrand <david@redhat.com>,
        David Rientjes <rientjes@google.com>,
        Muchun Song <muchun.song@linux.dev>,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com
Subject: Re: [RFC PATCH v2 0/5] hugetlb: parallelize hugetlb page init on boot
Message-ID: <20231212200653.GA7043@monkey>
References: <20231208025240.4744-1-gang.li@linux.dev>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20231208025240.4744-1-gang.li@linux.dev>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?XV7OkaXe8lewr0JWo4EozahHD+0ZSLbo2TKipbfQikHQorJHVUfdYK1H5aq6?=
 =?us-ascii?Q?qL7u+FkJUF5h5vTLYGclt/fMXcta9pgZMw75s6sGiPF6GEK0d6/3GEF96Wp3?=
 =?us-ascii?Q?XyBt5TKkwLcYPRiCXq0udkABG9rBRSlGLniRjHbjayVdGyeRnPiEZ34cjIDy?=
 =?us-ascii?Q?5UZNrAgWBrkPZifeQZv69YaJL7uoYXpUiDwkzPgL/EcF8KhbpcXHKiesPdaX?=
 =?us-ascii?Q?UStyLMcKGaXhYvaS5EipnNB3/I29YOIfofnJ2JB6VHHR9T9J3PDIO+ogekiT?=
 =?us-ascii?Q?EVkaB9RB9tVWoYO603/vnYKRQDNfGYA61qztiM6i+ldFkN7FrVlGE2beBrHS?=
 =?us-ascii?Q?Ar6hetH45CNQJl6kS2FCVTaQ+8LdGyd11KPDDSVvGa4nl/TxXFf/M9G+BPiP?=
 =?us-ascii?Q?qxpbeJdWYxK7HQixlwdTye5QHM49q7oBZV488N/8iWStG+g1dpa0dXj9phkB?=
 =?us-ascii?Q?gTdar4oJXglRrIX7i1SwUhPR2yv7r0L3AJKzbPndecOZznsvkZSa0LubElDL?=
 =?us-ascii?Q?9ahCRLtlYOKWWw9uX17sgMSRLdR5tK2X3dR28FzPdHpvI87W4q9u6ok0Gu9M?=
 =?us-ascii?Q?pBP6qd6ImLsiYqzS7lT8iAU/N+py77eRH0b6RKA6x7f83gpOFKKuhvch7fgs?=
 =?us-ascii?Q?eaGIaL76i4HAxYa/mBGZ0gQfOa3esKrDUpmOAFMWkuQEJJfeESOl/uc/2x60?=
 =?us-ascii?Q?jAjNQhLEILKnrl7M+TY8R7j1aQOvKlB7lhIZKy9HE7ntQYdma2w0GBDN8IhM?=
 =?us-ascii?Q?7j1knJufswSOU5M7uUcDJMldmVGjz8uH7TEFZM5o8j22fIthYUKgmfKPJob6?=
 =?us-ascii?Q?cUF9gutjWKN/f/tN4ANUVW05njmX0of8PMvK1mwiEi5a51M7UbCumEwaxAYF?=
 =?us-ascii?Q?bA7c0EJ66LgY9kbYZvnWCslZ+g2sXjTxU8ejtTJHaVVZ/YQIkvML1eta/uNd?=
 =?us-ascii?Q?7OCRCvO3G/OAf7p3Q3G6pM6PJtHOoecOIsZ13vorFLG6WGbmhWm3c6EQm/9P?=
 =?us-ascii?Q?NYc6dcElILqsKbxw2OvqvPW+ZLDsT0xAD3UVbJtQsfUHoRFX/dKUjTbsIJSq?=
 =?us-ascii?Q?CVqy9fQr+c2oxkpGQEsErj5H/kjpiiplnTy9xgbtJy8Y4EZLiYJsuo8X+tPe?=
 =?us-ascii?Q?OyahaxzEZuzhr5vidPiO58OVqPQnMmfauyaPSDunEaQ/bWaHoptkB7y4J30X?=
 =?us-ascii?Q?7TecM4qFrPmQvnmlIHAclP+J71tn4eL+jyiiTCA8StGm4/1gJaZLxG38SSn5?=
 =?us-ascii?Q?nniIElCrJRBiSF6jpkpU68kdr5/xNWHAfUfceiFE/+HO37afRuBv+C3Cqsd/?=
 =?us-ascii?Q?YibXNMxUVsadHV053nhPmyF4eLJv/G+nYNWePxHT3UDdNtcIDp0V1awKzaSH?=
 =?us-ascii?Q?VhsYrhScZTaHPOodxwp4eOMqsgwpvWx14bmWsUBspJRKd1BH59VtqlrnGKLr?=
 =?us-ascii?Q?zdG+MALvFesDPKPUb1J/aHcDubTtCEJJTBJkgNTGpJg1/nsn1jgQ3nBbWY8V?=
 =?us-ascii?Q?+kJc8yj7UoliRbVuuCD+UfpUzPHZDsn+KAfWFOkkkLeouo53fhhL24D8dNiy?=
 =?us-ascii?Q?AvaSJRdpFIqfIHSfwGApNC9vpSaR3A7GmIGgooFL?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?lYgMwTFd/CXd0MTuT8NbYXkTy80L+ecGMZ1Y1R7yVAjeno5kAm6bcctktjqn?=
 =?us-ascii?Q?c/yMvuADF2GthIYoyWurErLAxkumcbxZ6f9wwiO26PRxg2e8Klt69c08mrqf?=
 =?us-ascii?Q?6Cywsd0FFLeadODyO9Pq4ONMlpbY8MQT0DVcJqHAGeAhapYHM6bX+gfLtiZr?=
 =?us-ascii?Q?NHpqQBRiTlla2C33NVLSodB0KhKxnjT8nwfykgv0ebXHtL672T0pIG3+2kSx?=
 =?us-ascii?Q?+cwglA0xTj9OALkbBmLVNzSQ/GaFedUo6VG3HWoUnfXp5mWQ8hTJrpQ1Bf2l?=
 =?us-ascii?Q?jTgchiLWLyhbbJA86GYLuyxIBCWi7S3UXdN2/FXp0yIbgB5PyeH3CCN7flvs?=
 =?us-ascii?Q?zrnjAfsWUwuIbFz3fPyG7Kfw5LLiJVoWdSCB+jCXaDqE5qcKaLIHZB9Hib2n?=
 =?us-ascii?Q?uIZq48IBy6Ko0dCYK8T4P//idRqJf69gvJUcLcxfl/wFtTPbaqbFRDuS8ZnS?=
 =?us-ascii?Q?4YvT3ZG57AEc5Bz2ea+1Ev/XAYyCQ4kGUgpjMI0ZwzYyk0BWUjYt6jLxnz6r?=
 =?us-ascii?Q?bavPljUTEdjkuBNxZ1lltrO/Orz2vFcpHzWz8gjfuCTsIr/td0H/04qUo8vn?=
 =?us-ascii?Q?w7vJGxRSPaH22TgbDrH9l6ICGvufbGbZhgx2WGmSXTsqr8NwY7q1whh16X55?=
 =?us-ascii?Q?/7nef+FeIR0VSJ824MJMG2U3oI/Ugtpf9rHm48rbDMWozmjS4x0lvncp2yHf?=
 =?us-ascii?Q?RcQVUt4fsFsoRdCxaMfM3ODzvAvnmsIDWsIFCa0xlFLEVXgsqhNFe+t66OjR?=
 =?us-ascii?Q?OpU32vrR8+c24D23Kxw8vHHylKPaTrrev9Eg4gWMqYCvY17/yw9cufUR8FpS?=
 =?us-ascii?Q?1qfGZQRp8PJyJUDcNd2PPTjnqn/yVpAEtfBW6koPYelz/uIjdN5dZI7vnI2Y?=
 =?us-ascii?Q?j6gjMrM3hrs01P7rbg3hRwKWiNtYxbGsTO0awph1tgvb1BEoIiU90i3+E534?=
 =?us-ascii?Q?JkDTgpNizEijrcT3rN1o3Q=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2d87eb4e-039f-4e24-87ee-08dbfb4de3a2
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 20:06:55.6238
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CYVu7jGhou8wPbO8P8AG4BZAsG9aMxAnrt/ATdmKmst3YzOkUOMQnsYPtN/obEOyhMxemDvb6ARZp9GO8ykqZQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7126
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-12_12,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 suspectscore=0
 phishscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312120154
X-Proofpoint-ORIG-GUID: cnFCGbhYo_z7IQ4Z19Siv7pg6VP65Vq4
X-Proofpoint-GUID: cnFCGbhYo_z7IQ4Z19Siv7pg6VP65Vq4
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 12 Dec 2023 12:07:37 -0800 (PST)

On 12/08/23 10:52, Gang Li wrote:
> Hi all, hugetlb init parallelization has now been updated to v2.

Thanks for your efforts, and sorry for my late comments.

> To David Hildenbrand: padata multithread utilities has been used to reduce
> code complexity.
> 
> To David Rientjes: The patch for measuring time will be separately included
> in the reply. Please test during your free time, thanks.
> 
> # Introduction
> Hugetlb initialization during boot takes up a considerable amount of time.
> For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2
> seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel
> host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot
> time. This is a noteworthy figure.

One issue to be concerned with is hugetlb page allocation on systems with
unbalanced numa node memory.  Commit f60858f9d327 ("hugetlbfs: don't retry
when pool page allocations start to fail") was added to deal with issues
reported on such systems.  So, users are certainly using hugetlb pages
on systems with imbalances.

If performing allocations in parallel, I believe we would want the total
number of hugetlb pages allocated to be the same as today.  For example,
consider a simple 2 node system with 16GB total memory:
node 0:  2GB
node 1: 14GB

With today's code, allocating 6656 2MB pages via the kernel command line
results in:
node 0:  924 pages
node 1: 5732 pages
total:  6656 pages

With code to parallel allocations in this series:
node 0:  924 pages
node 1: 1547 pages
total:  2471 pages
-- 
Mike Kravetz