Received: by 2002:a05:7412:8d10:b0:f3:1519:9f41 with SMTP id bj16csp1765331rdb; Thu, 7 Dec 2023 08:13:01 -0800 (PST) X-Google-Smtp-Source: AGHT+IFdwbbtNOOONsbJ0IctdC/NiCBiEQczpxdCHh0XBCteoeKUUoL6B+dg2CV1UE3ICm+pxP2P X-Received: by 2002:a17:903:41c8:b0:1d0:7018:db36 with SMTP id u8-20020a17090341c800b001d07018db36mr2530683ple.104.1701965581444; Thu, 07 Dec 2023 08:13:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701965581; cv=none; d=google.com; s=arc-20160816; b=Yqfw8QM86xG201MFDkRhoK6VglUn6kY7RfegWpXftBpQ5Lgl3pr24FHGQ5i7kkW9B9 zHy24G38l+UPDUVSFTbyYMu9ymLdGUEUe7bUYydUV7GOawU2Vn3NnkHmjIlr3LAb+DZg MmYWMf0IUjOMr/yBliKLj+iD3sxDz8hNCVMZsUGBes27mZscYQYqIPZnKjxbzvdCDO9A DDJkBxjJWHykWCuKbeV9sYRedLMhh17OJQxSdpML5Cevv2rRCfSlbJFZGy+GmaUriyoS 8otrJVaTpLCZCoI9g7UMolKlUGOwBxjoKLMIjzwGzkzIX3dfekMrN5IGM59zgPZPDAq9 S54Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=CKQWUKL+0JPqdhCV9REIR79rPUrpl9LIOAnBol1yPV8=; fh=q/mRw8BjqfXMwBZlJBUekkoSHkVT4tbxFKy9NNgF+Vc=; b=WQ4gseTuemMZVA2+sqYwKrRktPXu96A31drsffe1hLtSX6McwI4vIIcqJLpOA0r8HI 5mP9q4GBnDluQoDuApsKMbI2TM7lwatwwm2YcMxHuM7NWk1/7+SOPVJhPIJK6EtueJyP Gzha5y4SkI7lbIUc5DCLg+hrIlTrY7xhLQaYQpA1YY5pAT9wyMhsMVlFHVF0I++YThzr DtnRGX/x+mEnlXmk3mtWGRL2yIIn36drboHoA3eaW7JCnlmzyQzWjSvsSxOdOF4c6qrT T7fJeMGRbtMEVBhumFI+TenH0AAGb1/EDB74NYC7N0kuvHB/2FoV6vfNa1lNtnOU6jXn wCzw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id s13-20020a170902b18d00b001d080c0e21fsi1284607plr.632.2023.12.07.08.13.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 08:13:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 9CE80819D9E6; Thu, 7 Dec 2023 08:12:49 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232960AbjLGQMY (ORCPT + 99 others); Thu, 7 Dec 2023 11:12:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232083AbjLGQMX (ORCPT ); Thu, 7 Dec 2023 11:12:23 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9365E10F3 for ; Thu, 7 Dec 2023 08:12:28 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7AFBF153B; Thu, 7 Dec 2023 08:13:13 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9732D3F762; Thu, 7 Dec 2023 08:12:24 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , John Hubbard , David Rientjes , Vlastimil Babka , Hugh Dickins , Kefeng Wang , Barry Song <21cnbao@gmail.com>, Alistair Popple Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 00/10] Multi-size THP for anonymous memory Date: Thu, 7 Dec 2023 16:12:01 +0000 Message-Id: <20231207161211.2374093-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 07 Dec 2023 08:12:50 -0800 (PST) Hi All, This is v9 (and hopefully the last) of a series to implement multi-size THP (mTHP) for anonymous memory (previously called "small-sized THP" and "large anonymous folios"). The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults: 1) Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel overhead. This should benefit all architectures. 2) Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch). This version incorporates David's feedback on the core patches (#3, #4) and adds some RB and TB tags (see change log for details). By default, the existing behaviour (and performance) is maintained. The user must explicitly enable multi-size THP to see the performance benefit. This is done via a new sysfs interface (as recommended by David Hildenbrand - thanks to David for the suggestion)! This interface is inspired by the existing per-hugepage-size sysfs interface used by hugetlb, provides full backwards compatibility with the existing PMD-size THP interface, and provides a base for future extensibility. See [9] for detailed discussion of the interface. This series is based on mm-unstable (715b67adf4c8). Prerequisites ============= I'm removing this section on the basis that I don't believe what we were previously calling prerequisites are really prerequisites anymore. We originally defined them when mTHP was a compile-time feature. There is now a runtime control to opt-in to mTHP; when disabled, correctness and performance are as before. When enabled, the code is still correct/robust, but in the absence of the one remaining item (compaction) there may be a performance impact in some corners. See the old list in the v8 cover letter at [8]. And a longer explanation of my thinking here [10]. SUMMARY: I don't think we should hold this series up, waiting for the items on the prerequisites list. I believe this series should be ready now so hopefully can be added to mm-unstable for some testing, then fingers crossed for v6.8. Testing ======= The series includes patches for mm selftests to enlighten the cow and khugepaged tests to explicitly test with multi-size THP, in the same way that PMD-sized THP is tested. The new tests all pass, and no regressions are observed in the mm selftest suite. I've also run my usual kernel compilation and java script benchmarks without any issues. Refer to my performance numbers posted with v6 [6]. (These are for multi-size THP only - they do not include the arm64 contpte follow-on series). John Hubbard at Nvidia has indicated dramatic 10x performance improvements for some workloads at [11]. (Observed using v6 of this series as well as the arm64 contpte series). Kefeng Wang at Huawei has also indicated he sees improvements at [12] although there are some latency regressions also. I've also checked that there is no regression in the write fault path when mTHP is disabled using a microbenchmark. I ran it for a baseline kernel, as well as v8 and v9. I repeated on Ampere Altra (bare metal) and Apple M2 (VM): | | m2 vm | altra | |--------------|---------------------|---------------------| | kernel | mean | std_rel | mean | std_rel | |--------------|----------|----------|----------|----------| | baseline | 0.000% | 0.341% | 0.000% | 3.581% | | anonfolio-v8 | 0.005% | 0.272% | 5.068% | 1.128% | | anonfolio-v9 | -0.013% | 0.442% | 0.107% | 1.788% | There is no measurable difference on M2, but altra has a slow down in v8 which is fixed in v9 by moving the THP order check to be inline within thp_vma_allowable_orders(), as suggested by David. Changes since v8 [8] ==================== - Added various Reviewed-by/Tested-by tags (Barry, David, Kefeng, John) - Patch 3: - Renamed first_order() -> highest_order() (David) - Made helpers for thp_vma_suitable_orders() thp_vma_allowable_orders() that take a single unencoded order parameter: thp_vma_suitable_order() and thp_vma_allowable_order(), and use them to aid readability (David) - Split thp_vma_allowable_orders() into an order-0 fast-path inline and slow-path __thp_vma_allowable_orders() part (David) - Added spin lock to serialize changes to huge_anon_orders_* fields to prevent possibility of clearing all bits when threads are racing (David) - Patch 4: - Pass address of faulting page (not start of folio) to clear_huge_page() - Reverse xmas tree for variable lists (David) - Added unlikely() for uffd check (David) - tidied up a local variable in alloc_anon_folio() (David) - Separated update_mmu_tlb() handling for nr_pages == 1 vs > 1 (David) Changes since v7 [7] ==================== - Renamed "small-sized THP" -> "multi-size THP" in commit logs - Added various Reviewed-by/Tested-by tags (Barry, David, Alistair) - Patch 3: - Fine-tuned transhuge documentation multi-size THP (JohnH) - Converted hugepage_global_enabled() and hugepage_global_always() macros to static inline functions (JohnH) - Renamed hugepage_vma_check() to thp_vma_allowable_orders() (JohnH) - Renamed transhuge_vma_suitable() to thp_vma_suitable_orders() (JohnH) - Renamed "global" enabled sysfs file option to "inherit" (JohnH) - Patch 9: - cow selftest: Renamed param size -> thpsize (David) - cow selftest: Changed test fail to assert() (David) - cow selftest: Log PMD size separately from all the supported THP sizes (David) - Patch 10: - cow selftest: No longer special case pmdsize; keep all THP sizes in thpsizes[] Changes since v6 [6] ==================== - Refactored vmf_pte_range_changed() to remove uffd special-case (suggested by JohnH) - Dropped accounting patch (#3 in v6) (suggested by DavidH) - Continue to account *PMD-sized* THP only for now - Can add more counters in future if needed - Page cache large folios haven't needed any new counters yet - Pivot to sysfs ABI proposed by DavidH - per-size directories in a similar shape to that used by hugetlb - Dropped "recommend" keyword patch (#6 in v6) (suggested by DavidH, Yu Zhou) - For now, users need to understand implicitly which sizes are beneficial to their HW/SW - Dropped arch_wants_pte_order() patch (#7 in v6) - No longer needed due to dropping patch "recommend" keyword patch - Enlightened khugepaged mm selftest to explicitly test with small-size THP - Scrubbed commit logs to use "small-sized THP" consistently (suggested by DavidH) Changes since v5 [5] ==================== - Added accounting for PTE-mapped THPs (patch 3) - Added runtime control mechanism via sysfs as extension to THP (patch 4) - Minor refactoring of alloc_anon_folio() to integrate with runtime controls - Stripped out hardcoded policy for allocation order; its now all user space controlled (although user space can request "recommend" which will configure the HW-preferred order) Changes since v4 [4] ==================== - Removed "arm64: mm: Override arch_wants_pte_order()" patch; arm64 now uses the default order-3 size. I have moved this patch over to the contpte series. - Added "mm: Allow deferred splitting of arbitrary large anon folios" back into series. I originally removed this at v2 to add to a separate series, but that series has transformed significantly and it no longer fits, so bringing it back here. - Reintroduced dependency on set_ptes(); Originally dropped this at v2, but set_ptes() is in mm-unstable now. - Updated policy for when to allocate LAF; only fallback to order-0 if MADV_NOHUGEPAGE is present or if THP disabled via prctl; no longer rely on sysfs's never/madvise/always knob. - Fallback to order-0 whenever uffd is armed for the vma, not just when uffd-wp is set on the pte. - alloc_anon_folio() now returns `struct folio *`, where errors are encoded with ERR_PTR(). The last 3 changes were proposed by Yu Zhao - thanks! Changes since v3 [3] ==================== - Renamed feature from FLEXIBLE_THP to LARGE_ANON_FOLIO. - Removed `flexthp_unhinted_max` boot parameter. Discussion concluded that a sysctl is preferable but we will wait until real workload needs it. - Fixed uninitialized `addr` on read fault path in do_anonymous_page(). - Added mm selftests for large anon folios in cow test suite. Changes since v2 [2] ==================== - Dropped commit "Allow deferred splitting of arbitrary large anon folios" - Huang, Ying suggested the "batch zap" work (which I dropped from this series after v1) is a prerequisite for merging FLXEIBLE_THP, so I've moved the deferred split patch to a separate series along with the batch zap changes. I plan to submit this series early next week. - Changed folio order fallback policy - We no longer iterate from preferred to 0 looking for acceptable policy - Instead we iterate through preferred, PAGE_ALLOC_COSTLY_ORDER and 0 only - Removed vma parameter from arch_wants_pte_order() - Added command line parameter `flexthp_unhinted_max` - clamps preferred order when vma hasn't explicitly opted-in to THP - Never allocate large folio for MADV_NOHUGEPAGE vma (or when THP is disabled for process or system). - Simplified implementation and integration with do_anonymous_page() - Removed dependency on set_ptes() Changes since v1 [1] ==================== - removed changes to arch-dependent vma_alloc_zeroed_movable_folio() - replaced with arch-independent alloc_anon_folio() - follows THP allocation approach - no longer retry with intermediate orders if allocation fails - fallback directly to order-0 - remove folio_add_new_anon_rmap_range() patch - instead add its new functionality to folio_add_new_anon_rmap() - remove batch-zap pte mappings optimization patch - remove enabler folio_remove_rmap_range() patch too - These offer real perf improvement so will submit separately - simplify Kconfig - single FLEXIBLE_THP option, which is independent of arch - depends on TRANSPARENT_HUGEPAGE - when enabled default to max anon folio size of 64K unless arch explicitly overrides - simplify changes to do_anonymous_page(): - no more retry loop [1] https://lore.kernel.org/linux-mm/20230626171430.3167004-1-ryan.roberts@arm.com/ [2] https://lore.kernel.org/linux-mm/20230703135330.1865927-1-ryan.roberts@arm.com/ [3] https://lore.kernel.org/linux-mm/20230714160407.4142030-1-ryan.roberts@arm.com/ [4] https://lore.kernel.org/linux-mm/20230726095146.2826796-1-ryan.roberts@arm.com/ [5] https://lore.kernel.org/linux-mm/20230810142942.3169679-1-ryan.roberts@arm.com/ [6] https://lore.kernel.org/linux-mm/20230929114421.3761121-1-ryan.roberts@arm.com/ [7] https://lore.kernel.org/linux-mm/20231122162950.3854897-1-ryan.roberts@arm.com/ [8] https://lore.kernel.org/linux-mm/20231204102027.57185-1-ryan.roberts@arm.com/ [9] https://lore.kernel.org/linux-mm/6d89fdc9-ef55-d44e-bf12-fafff318aef8@redhat.com/ [10] https://lore.kernel.org/linux-mm/2de0617e-d1d7-49ec-9cb8-206eaf37caed@arm.com/ [11] https://lore.kernel.org/linux-mm/c507308d-bdd4-5f9e-d4ff-e96e4520be85@nvidia.com/ [12] https://lore.kernel.org/linux-mm/b8f5a47a-af1e-44ed-a89b-460d0be56d2c@huawei.com/ Thanks, Ryan Ryan Roberts (10): mm: Allow deferred splitting of arbitrary anon large folios mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() mm: thp: Introduce multi-size THP sysfs interface mm: thp: Support allocation of anonymous multi-size THP selftests/mm/kugepaged: Restore thp settings at exit selftests/mm: Factor out thp settings management selftests/mm: Support multi-size THP interface in thp_settings selftests/mm/khugepaged: Enlighten for multi-size THP selftests/mm/cow: Generalize do_run_with_thp() helper selftests/mm/cow: Add tests for anonymous multi-size THP Documentation/admin-guide/mm/transhuge.rst | 97 ++++- Documentation/filesystems/proc.rst | 6 +- fs/proc/task_mmu.c | 3 +- include/linux/huge_mm.h | 183 +++++++-- mm/huge_memory.c | 231 ++++++++++-- mm/khugepaged.c | 20 +- mm/memory.c | 117 +++++- mm/page_vma_mapped.c | 3 +- mm/rmap.c | 32 +- tools/testing/selftests/mm/Makefile | 4 +- tools/testing/selftests/mm/cow.c | 185 +++++++--- tools/testing/selftests/mm/khugepaged.c | 410 ++++----------------- tools/testing/selftests/mm/run_vmtests.sh | 2 + tools/testing/selftests/mm/thp_settings.c | 349 ++++++++++++++++++ tools/testing/selftests/mm/thp_settings.h | 80 ++++ 15 files changed, 1218 insertions(+), 504 deletions(-) create mode 100644 tools/testing/selftests/mm/thp_settings.c create mode 100644 tools/testing/selftests/mm/thp_settings.h -- 2.25.1