Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Joao Martins <joao.m.martins@oracle.com>,
        Konrad Dybcio <konradybcio@kernel.org>,
        Oscar Salvador <osalvador@suse.de>,
        David Hildenbrand <david@redhat.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        David Rientjes <rientjes@google.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        Barry Song <21cnbao@gmail.com>, Michal Hocko <mhocko@suse.com>,
        Matthew Wilcox <willy@infradead.org>,
        Xiongchun Duan <duanxiongchun@bytedance.com>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Usama Arif <usama.arif@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>,
        James Houghton <jthoughton@google.com>
Subject: [PATCH v8 1/8] hugetlb: optimize update_and_free_pages_bulk to avoid lock cycles
Date:   Wed, 18 Oct 2023 19:31:03 -0700
Message-ID: <20231019023113.345257-2-mike.kravetz@oracle.com>
In-Reply-To: <20231019023113.345257-1-mike.kravetz@oracle.com>
References: <20231019023113.345257-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fDX8WngwI5tItkRa0jsEXfz2RznQjt4vANh2q3dj8ewmTU7d/aj6aB9CNdRg?=
 =?us-ascii?Q?mmCgenBkSxEY/elEnJh0faJBn/w1RUZkG+evHtHcLCPAcAJ/RZdAfOqctvUC?=
 =?us-ascii?Q?Epg0wmUhY7ME/RocfUb0Y2KmPLfMODXQ9KvAPhhvxiIYrCk7tNxjWYPBarOk?=
 =?us-ascii?Q?HMte00sBQQ7BAA+jfeHw7v8Siplr0hZ8rvWr8Vi9cjXPNdlkASrQPTv4uW+a?=
 =?us-ascii?Q?8SHr6ooS2gw32EHP01lOTfdqBHIReiRdu1PKav6aBLd0jr92GR1b+I1W6+0r?=
 =?us-ascii?Q?oLkxDDaLWHcDqxGmggR1HAYUxpZOfBnxe71KAx6ZcvoYAdTH3DRKPFufaWNc?=
 =?us-ascii?Q?zcCBX8wApPi2yO5gfqprAy3Zq/VqMCJE6tRWwcYuxge/yKz/Q7Y3i0w/1xdd?=
 =?us-ascii?Q?RnWdRUFwWfUae+FViDStOzSYtouXeNyJC4CI09XRA+FZlZ5Ku/w2Lm+kH4vE?=
 =?us-ascii?Q?UFoBFFqP6cliwc3iwF3rzulGPfkCTM6/DEdLpfrUGaQp5EJyhQqSvCzhJc+b?=
 =?us-ascii?Q?VQ92+tm5d+ybFQmYqKtYaLN0ug3ekjgo9IhMn0FYIED2z1ud5JIvM96RVC2T?=
 =?us-ascii?Q?CBA39xDcKB+pbTU7eK4Hd15mb4Wb1afSsRqIyVaKDlgNA/RYFffLFbkK/Wtv?=
 =?us-ascii?Q?pKq4WM91RsmyO7IbyVpXxzIRCJOS/esReAMayde0PnWY1iMo8q6VyBvukbaE?=
 =?us-ascii?Q?t3XkC2o3C7uKZxdJHugVDt06EFMsMbBg7HsYrXHBNVfTowYd/XtkJHCNvTS7?=
 =?us-ascii?Q?/3h0DG1N39X4OLP/4PHQBxAoCSinZklAR230mJaYv+90YSrQ4+dX6nKwwA0F?=
 =?us-ascii?Q?beOLPYsp/jQJTvoPfPRK9BR1nbLWNga7Y4MgJc9z4FUA4lz2kpliEs8Zg1rB?=
 =?us-ascii?Q?9vqvPMXaDbLC6MmEXdHXMWR1YWUgk1/pqiQqthitTnYf+4YQz5OuOKn4j+xT?=
 =?us-ascii?Q?RVyqO9uFP8B69pjp4ySJLdcyYaoU3KBCKABNx54Ay8VxastoGlwGHOhOgB9h?=
 =?us-ascii?Q?AhI1+NZAXzSkShq28ujZCLhHZ/3DMOuYyknYjJVfyLVByjpBGh6fdpcZUmz9?=
 =?us-ascii?Q?mbnP/hedF5s/D84yD6d/HG1awcPkX8nLCGhYl4EFCmJwu3gN1D/4B1Z4Tw+J?=
 =?us-ascii?Q?NVubrwC43B5jqLnXmvelEU6tUgZaDAB5FuASdJnU1VzNXXxclkamL0chSuGJ?=
 =?us-ascii?Q?AKY6Dc19x2wNDfTUD+Vu4+X275yWIFlq/EdG/3flt5/D9mnXgKPPaO934adf?=
 =?us-ascii?Q?yN1O5jZSxqfuXXn0bpzjHBKsur8gCsQwqs8xAy6OkAT6tJDajdWw18phNe0N?=
 =?us-ascii?Q?sOT9KLSJYXfOYInlahIYw7F3xXYm8152xmE0qcW93hiLrIxAyjFGPlNUtT0l?=
 =?us-ascii?Q?kHghMwNc1EVoVkh7vcZiaAjUeDbDP/zHsjJ2etT1fWxrMDQb4ZDdkhZf9Yst?=
 =?us-ascii?Q?CyvkDpbSknMUapHxkwbpUcjVatVKVjsI06x7RCOKwggdEty5yN1y79BlWac1?=
 =?us-ascii?Q?2CA8gXbKg7kq7GLk8ETUu+RPFKo03LfQMHMUCKlXEC6fBOFw3I/XQuf8KqlU?=
 =?us-ascii?Q?QRIE46+MK41cQ08aa+AeNNGo2PUb5FUxg7KvzI/Wr0+HSX/xE1Ch8Y99QcIH?=
 =?us-ascii?Q?nQ=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?u05OtFiN+b2Z/V5/d6o9tBg2EYRGdopbGya8sv3u8cowCnNv03RIqY8C2KDG?=
 =?us-ascii?Q?kF7Sw52UHlFigtowWI56Em3znndJ8UKqbqPIzeHBkln8nX8BZAtpFfqezSRm?=
 =?us-ascii?Q?5vWH6dgj7NskUmy8SZOA6fV0b9UMy7GCjBT8TLVwrvQb+iravV4df3eeAVyK?=
 =?us-ascii?Q?FIMFcti+PYmTsQ2zwlgHiCm1rc/mLdq4887gDD1Srgxg/8HfZ7d+iKrSDIaW?=
 =?us-ascii?Q?B739gIp+3J5eOdZBaPAp4/B0IJNGbb2RSyRoE7AZFvnMFmEpFsIZSiBYCvK2?=
 =?us-ascii?Q?0ERQAaV3oJ4p//EjbkYORNh1thw2/OYjev+WUIkStMAHILuwgZ0D8jfHNvSQ?=
 =?us-ascii?Q?HEkfjVs/KPDQmWjKBbUSZ5asg0d/FEmEzFBQX0Om6/AfkWCVjmRYroVPc59M?=
 =?us-ascii?Q?hbFPHUC6YHaFRyVFfGd6EmdPSNqllRKi4KI8sHVq4IoX0sm6ePMbZoKkvvP6?=
 =?us-ascii?Q?HbPMoT/TC2A/hYhzzcNIKd78pjZNLECabWQ/dXI+T4gpW34dS3lP7FmUP9DC?=
 =?us-ascii?Q?lrYeexkixlH8BgPuUfFCeSHCtNCEL/BDMXAW9lASYnT6Cg3VIPisIpFa9auA?=
 =?us-ascii?Q?ePZQYhL7qAayTSnHG3Pvjya9oH6BnMOBLpouONFTuabhcHrIC3EJlUfFev05?=
 =?us-ascii?Q?RWBZu8jze7KOnYFmgfXWmYyTmZXDMMV8A2+d+4r+KYPp1PxEGjLPyq/brIDW?=
 =?us-ascii?Q?MqrafgoKMdCX8r+lAzZFc0BnQaWCeaNQz3WqVnfttyBo5nH7v0hGrugTiVtd?=
 =?us-ascii?Q?A/BFOUlvP6UwgpfVwCqIrI3Z0Iz1t7K5zunZhvdDIIDZ5jxoi6C+kb2Z4yN1?=
 =?us-ascii?Q?lP480eNqx5N2W1BPR3Cw99OX32CSB+BQaJgYflXw+OzmiSYQ3a9qt61zNkd5?=
 =?us-ascii?Q?+9z6oCaae5jQdBXDwcSFJQXLztg8SwpEQiyy2PBnp630JyjhMlUFIpxeVquV?=
 =?us-ascii?Q?cnU5n9o+QB/ZyfgDiWgT34lKFtIwWDW+IhdYKiECqDauaq6+rAcBfSn3/8jt?=
 =?us-ascii?Q?xxQQ4NsrF0xncITXzzDEGYcfdRRX769fjgVLnjRF198QqVO5Zqlsmfmseba0?=
 =?us-ascii?Q?o0WM0nmZFy7FW39+8uwVFFs0isxpTXsBkWHx9b+At3qSMbf8vZHlC92q95qO?=
 =?us-ascii?Q?1yvH2iPZ1nSq5GLYrhUIHPNAeqTKHzk26bqMS73uj3m01Pt5Tv1TcULYed8G?=
 =?us-ascii?Q?q3WMtyJ/juAYN09k?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3497091d-d510-4766-32d6-08dbd04b7a93
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2023 02:31:20.1867
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: e0La7n9nhs6MV2VIaz83dp3QI3MlRcpJngx4sj4Et+DTJMf9+zb7beJJszoYD1PFOthBchx9vnrp7lARveidkg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB7123
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-19_02,2023-10-18_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 phishscore=0 spamscore=0 suspectscore=0 malwarescore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2309180000 definitions=main-2310190019
X-Proofpoint-GUID: eVSFDYN1w36MvDUeeE3XEksew9_kiHYQ
X-Proofpoint-ORIG-GUID: eVSFDYN1w36MvDUeeE3XEksew9_kiHYQ
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 18 Oct 2023 19:32:28 -0700 (PDT)

update_and_free_pages_bulk is designed to free a list of hugetlb pages
back to their associated lower level allocators.  This may require
allocating vmemmmap pages associated with each hugetlb page.  The
hugetlb page destructor must be changed before pages are freed to lower
level allocators.  However, the destructor must be changed under the
hugetlb lock.  This means there is potentially one lock cycle per page.

Minimize the number of lock cycles in update_and_free_pages_bulk by:
1) allocating necessary vmemmap for all hugetlb pages on the list
2) take hugetlb lock and clear destructor for all pages on the list
3) free all pages on list back to low level allocators

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
Reviewed-by: Muchun Song <songmuchun@bytedance.com>
Acked-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index da6f85b7db88..b839080a2a6b 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1862,7 +1862,46 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio,
 static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list)
 {
 	struct folio *folio, *t_folio;
+	bool clear_dtor = false;
 
+	/*
+	 * First allocate required vmemmmap (if necessary) for all folios on
+	 * list.  If vmemmap can not be allocated, we can not free folio to
+	 * lower level allocator, so add back as hugetlb surplus page.
+	 * add_hugetlb_folio() removes the page from THIS list.
+	 * Use clear_dtor to note if vmemmap was successfully allocated for
+	 * ANY page on the list.
+	 */
+	list_for_each_entry_safe(folio, t_folio, list, lru) {
+		if (folio_test_hugetlb_vmemmap_optimized(folio)) {
+			if (hugetlb_vmemmap_restore(h, &folio->page)) {
+				spin_lock_irq(&hugetlb_lock);
+				add_hugetlb_folio(h, folio, true);
+				spin_unlock_irq(&hugetlb_lock);
+			} else
+				clear_dtor = true;
+		}
+	}
+
+	/*
+	 * If vmemmmap allocation was performed on any folio above, take lock
+	 * to clear destructor of all folios on list.  This avoids the need to
+	 * lock/unlock for each individual folio.
+	 * The assumption is vmemmap allocation was performed on all or none
+	 * of the folios on the list.  This is true expect in VERY rare cases.
+	 */
+	if (clear_dtor) {
+		spin_lock_irq(&hugetlb_lock);
+		list_for_each_entry(folio, list, lru)
+			__clear_hugetlb_destructor(h, folio);
+		spin_unlock_irq(&hugetlb_lock);
+	}
+
+	/*
+	 * Free folios back to low level allocators.  vmemmap and destructors
+	 * were taken care of above, so update_and_free_hugetlb_folio will
+	 * not need to take hugetlb lock.
+	 */
 	list_for_each_entry_safe(folio, t_folio, list, lru) {
 		update_and_free_hugetlb_folio(h, folio, false);
 		cond_resched();
-- 
2.41.0