Received: by 2002:a05:6a10:9afc:0:0:0:0 with SMTP id t28csp1088593pxm; Wed, 23 Feb 2022 17:50:21 -0800 (PST) X-Google-Smtp-Source: ABdhPJxp8FnjC2IB39FDcsYzmN2jee/GGNcu8/PxIN7CPNFVFW7W7Pn/z/IxxJxXAO9fkD4jsFNV X-Received: by 2002:a05:6a00:27a4:b0:4f3:9654:2673 with SMTP id bd36-20020a056a0027a400b004f396542673mr616253pfb.41.1645667421077; Wed, 23 Feb 2022 17:50:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645667421; cv=none; d=google.com; s=arc-20160816; b=Sw01jDda0PJaTG0CHWjMdoO3sXfaV+lnZvyJSgXrMy9zTzajgfs/qHVMVTMwRLuO8G q+MHn5LwG9VImGEJ/wK+UHLyoaFe4cCcxUa5X9SSNw1bVuy7Juhb65le9CvNNdbd2kkS kb77vf3ilr2bUWbWIQlurfexNiScV+2ZtcCj7O7ZKH212egTvst8Bj560bvA1YsxxwLU Tuk11PIqhODtNQTr3VviUJdvJ2qbd+RIpxvkVJ7r5TPA9HpunHkdl2LGeTXFOrPbD+nM TsI1P1pxN4kR8UTrPN3QoH3iK8h7ahUTSom4i88NEZR5PGdMWiIGJ8qbpvC7HyfPfIxt r2CA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:references:mime-version :message-id:in-reply-to:date:dkim-signature; bh=PIzleJhvx1e4in5SU2cz59acEkt7I4gZXxP/VePrhcs=; b=WKo6aYdLYhnoAselFpKQfkYVk80nh4Tda4y9MzTAACA9t8tfAORQ9fu4IPnkcJe+NI mmtLosY0k1v5e4VJZ5oiapuhoPlvH5Cjtq4FwBNnJ6Kj2fn1NEwTgzyME7JT6wmbd315 lSMDhOclPVz+Dr9yQ8HW8cbQgppAXtsIubBmiyyLOdNGDrNzYQniOd+CdLqY1pnAF2Wm 3mwAocD6GxVXSFLfXUQ3e6JeBdM8F6TvDilUZuCCuzZrmJHUhM4QInqXPOZsT3GnnZf+ WUN7hCK345Ryys0vddkKPuRqIUwPo1KiAol4OXLAhTdRsqmx+uwTkabCCZShU/S9m4xm a+fw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=UOhpXDHC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id bh14si1301332pgb.425.2022.02.23.17.50.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Feb 2022 17:50:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=UOhpXDHC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9409F1B0C5C; Wed, 23 Feb 2022 17:22:29 -0800 (PST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238151AbiBWFYf (ORCPT + 99 others); Wed, 23 Feb 2022 00:24:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238123AbiBWFY2 (ORCPT ); Wed, 23 Feb 2022 00:24:28 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE3416A047 for ; Tue, 22 Feb 2022 21:23:59 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2d306e372e5so163250677b3.5 for ; Tue, 22 Feb 2022 21:23:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=PIzleJhvx1e4in5SU2cz59acEkt7I4gZXxP/VePrhcs=; b=UOhpXDHCnae+DRnauRxZtdjiTln+VoGZhoV1LC2HwZ0wE6hCMEDx6ZsM5m4zhmzU8C zHAJJxzPwGGULan8LLoI7C3OSmGzlkgZIkeoNkliXHKxR2zxRtRuDwE2nmiSS1c4BXRL MiuUyWSM6+TUYE5RqqQekigHUWwoMR7es7zfFE2KzZ5ooe/XoTE9TQgA48klkJO1uSey E656AcQf7rLEbAA6uzuZiKaqjksL8iwXpjTOMvdQBp3iyrK4xwvnjKTRQ5lhnl0vyEeu pioGnqi8o2SeqNRitqaZjawpEPOB4cWAR7yNk2BDoyC4/ldrIIsdHyj3n/nSEdlK5M0a gypQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PIzleJhvx1e4in5SU2cz59acEkt7I4gZXxP/VePrhcs=; b=Ipj/5KE1k0ES7yurFMTzfPS58abKskOdWc/E+l4HHvqsBBsTK84cWq2bnbXX4SWf1i UlZBmO0hL2qEv++npNlMW+aaz1VL7nwzmZbUhNaK9SoiF7Ei05604FVRinuxGpNn0p7j 4ZKitdEPKLzRr/vE5g+3s6x6u52ciN+H5HVBG8olpVqthm5AdTPXXwrs9u5zP+EXXcLe WuN29kfYlWT1t54xbxgoMgVFqGXzlyLCJjS/DecH0JKxPEwK1V0nwGoUjHXlIIhAhIMw +ZyjfO9oHksrDyYP4A+sIM5z221LmmhsxdedgVWcmVuHRqQj6VO7SEB8tNTafThFfQTu gAiA== X-Gm-Message-State: AOAM530O4HsYO4s97+Z9snFTcsil8+sxpxL/fV3iTuYa9vk0hFuX3cZH TzDUEh++Tkwbloqprj9DR6YN0UmZlEEn9shiY/t924P3BTexLTrn8PEv2dpPRrMlw1unAQzSPvC Jnhjr3J9e78uVU24jsQm1qKSEZce+u7OFQeT/m1qY4E77v3z8gz/fAOiZODXbcVXlrWUtyWWL X-Received: from js-desktop.svl.corp.google.com ([2620:15c:2cd:202:ccbe:5d15:e2e6:322]) (user=junaids job=sendgmr) by 2002:a0d:c607:0:b0:2ca:287c:6b6c with SMTP id i7-20020a0dc607000000b002ca287c6b6cmr28060793ywd.17.1645593839000; Tue, 22 Feb 2022 21:23:59 -0800 (PST) Date: Tue, 22 Feb 2022 21:21:42 -0800 In-Reply-To: <20220223052223.1202152-1-junaids@google.com> Message-Id: <20220223052223.1202152-7-junaids@google.com> Mime-Version: 1.0 References: <20220223052223.1202152-1-junaids@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [RFC PATCH 06/47] mm: asi: ASI page table allocation and free functions From: Junaid Shahid To: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, pjt@google.com, oweisse@google.com, alexandre.chartre@oracle.com, rppt@linux.ibm.com, dave.hansen@linux.intel.com, peterz@infradead.org, tglx@linutronix.de, luto@kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This adds custom allocation and free functions for ASI page tables. The alloc functions support allocating memory using different GFP reclaim flags, in order to be able to support non-sensitive allocations from both standard and atomic contexts. They also install the page tables locklessly, which makes it slightly simpler to handle non-sensitive allocations from interrupts/exceptions. The free functions recursively free the page tables when the ASI instance is being torn down. Signed-off-by: Junaid Shahid --- arch/x86/mm/asi.c | 109 +++++++++++++++++++++++++++++++++++++++- include/linux/pgtable.h | 3 ++ 2 files changed, 111 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 2453124f221d..40d772b2e2a8 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -60,6 +60,113 @@ void asi_unregister_class(int index) } EXPORT_SYMBOL_GPL(asi_unregister_class); +#ifndef mm_inc_nr_p4ds +#define mm_inc_nr_p4ds(mm) do {} while (false) +#endif + +#ifndef mm_dec_nr_p4ds +#define mm_dec_nr_p4ds(mm) do {} while (false) +#endif + +#define pte_offset pte_offset_kernel + +#define DEFINE_ASI_PGTBL_ALLOC(base, level) \ +static level##_t * asi_##level##_alloc(struct asi *asi, \ + base##_t *base, ulong addr, \ + gfp_t flags) \ +{ \ + if (unlikely(base##_none(*base))) { \ + ulong pgtbl = get_zeroed_page(flags); \ + phys_addr_t pgtbl_pa; \ + \ + if (pgtbl == 0) \ + return NULL; \ + \ + pgtbl_pa = __pa(pgtbl); \ + paravirt_alloc_##level(asi->mm, PHYS_PFN(pgtbl_pa)); \ + \ + if (cmpxchg((ulong *)base, 0, \ + pgtbl_pa | _PAGE_TABLE) == 0) { \ + mm_inc_nr_##level##s(asi->mm); \ + } else { \ + paravirt_release_##level(PHYS_PFN(pgtbl_pa)); \ + free_page(pgtbl); \ + } \ + \ + /* NOP on native. PV call on Xen. */ \ + set_##base(base, *base); \ + } \ + VM_BUG_ON(base##_large(*base)); \ + return level##_offset(base, addr); \ +} + +DEFINE_ASI_PGTBL_ALLOC(pgd, p4d) +DEFINE_ASI_PGTBL_ALLOC(p4d, pud) +DEFINE_ASI_PGTBL_ALLOC(pud, pmd) +DEFINE_ASI_PGTBL_ALLOC(pmd, pte) + +#define asi_free_dummy(asi, addr) +#define __pmd_free(mm, pmd) free_page((ulong)(pmd)) +#define pud_page_vaddr(pud) ((ulong)pud_pgtable(pud)) +#define p4d_page_vaddr(p4d) ((ulong)p4d_pgtable(p4d)) + +static inline unsigned long pte_page_vaddr(pte_t pte) +{ + return (unsigned long)__va(pte_val(pte) & PTE_PFN_MASK); +} + +#define DEFINE_ASI_PGTBL_FREE(level, LEVEL, next, free) \ +static void asi_free_##level(struct asi *asi, ulong pgtbl_addr) \ +{ \ + uint i; \ + level##_t *level = (level##_t *)pgtbl_addr; \ + \ + for (i = 0; i < PTRS_PER_##LEVEL; i++) { \ + ulong vaddr; \ + \ + if (level##_none(level[i])) \ + continue; \ + \ + vaddr = level##_page_vaddr(level[i]); \ + \ + if (!level##_leaf(level[i])) \ + asi_free_##next(asi, vaddr); \ + else \ + VM_WARN(true, "Lingering mapping in ASI %p at %lx",\ + asi, vaddr); \ + } \ + paravirt_release_##level(PHYS_PFN(__pa(pgtbl_addr))); \ + free(asi->mm, level); \ + mm_dec_nr_##level##s(asi->mm); \ +} + +DEFINE_ASI_PGTBL_FREE(pte, PTE, dummy, pte_free_kernel) +DEFINE_ASI_PGTBL_FREE(pmd, PMD, pte, __pmd_free) +DEFINE_ASI_PGTBL_FREE(pud, PUD, pmd, pud_free) +DEFINE_ASI_PGTBL_FREE(p4d, P4D, pud, p4d_free) + +static void asi_free_pgd_range(struct asi *asi, uint start, uint end) +{ + uint i; + + for (i = start; i < end; i++) + if (pgd_present(asi->pgd[i])) + asi_free_p4d(asi, (ulong)p4d_offset(asi->pgd + i, 0)); +} + +/* + * Free the page tables allocated for the given ASI instance. + * The caller must ensure that all the mappings have already been cleared + * and appropriate TLB flushes have been issued before calling this function. + */ +static void asi_free_pgd(struct asi *asi) +{ + VM_BUG_ON(asi->mm == &init_mm); + + asi_free_pgd_range(asi, KERNEL_PGD_BOUNDARY, PTRS_PER_PGD); + free_pages((ulong)asi->pgd, PGD_ALLOCATION_ORDER); +} + static int __init set_asi_param(char *str) { if (strcmp(str, "on") == 0) @@ -102,7 +209,7 @@ void asi_destroy(struct asi *asi) if (!boot_cpu_has(X86_FEATURE_ASI)) return; - free_pages((ulong)asi->pgd, PGD_ALLOCATION_ORDER); + asi_free_pgd(asi); memset(asi, 0, sizeof(struct asi)); } EXPORT_SYMBOL_GPL(asi_destroy); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e24d2c992b11..2fff17a939f0 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1593,6 +1593,9 @@ typedef unsigned int pgtbl_mod_mask; #ifndef pmd_leaf #define pmd_leaf(x) 0 #endif +#ifndef pte_leaf +#define pte_leaf(x) 1 +#endif #ifndef pgd_leaf_size #define pgd_leaf_size(x) (1ULL << PGDIR_SHIFT) -- 2.35.1.473.g83b2b277ed-goog