Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= <bjorn@kernel.org>
To:     Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        linux-riscv@lists.infradead.org
Cc:     =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= <bjorn@rivosinc.com>,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        David Hildenbrand <david@redhat.com>,
        Oscar Salvador <osalvador@suse.de>,
        virtualization@lists.linux-foundation.org, linux@rivosinc.com,
        Alexandre Ghiti <alexghiti@rivosinc.com>
Subject: [PATCH 7/7] riscv: mm: Pre-allocate vmalloc PGD leaves
Date:   Fri, 12 May 2023 16:57:37 +0200
Message-Id: <20230512145737.985671-8-bjorn@kernel.org>
In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org>
References: <20230512145737.985671-1-bjorn@kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

From: Björn Töpel <bjorn@rivosinc.com>

Instead of relying on vmalloc_fault() to synchronize the page-tables,
pre-allocate the PGD leaves of the vmalloc area. This is only enabled
if memory hot/add is enabled by the build.

Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
---
 arch/riscv/mm/fault.c | 7 ++++++-
 arch/riscv/mm/init.c  | 1 +
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 8685f85a7474..b61e279acd50 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -233,12 +233,17 @@ void handle_page_fault(struct pt_regs *regs)
 	 * Fault-in kernel-space virtual memory on-demand.
 	 * The 'reference' page table is init_mm.pgd.
 	 *
+	 * For memory hotplug enabled systems, the PGD entries are
+	 * pre-allocated, which avoids the need to synchronize
+	 * pgd/fault-in.
+	 *
 	 * NOTE! We MUST NOT take any locks for this case. We may
 	 * be in an interrupt or a critical region, and should
 	 * only copy the information from the master page table,
 	 * nothing more.
 	 */
-	if (unlikely((addr >= VMALLOC_START) && (addr < VMALLOC_END))) {
+	if (unlikely(!IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
+		     (addr >= VMALLOC_START) && (addr < VMALLOC_END))) {
 		vmalloc_fault(regs, code, addr);
 		return;
 	}
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index a468708d1e1c..fd5a6d3fe182 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -236,6 +236,7 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon
 static void __init prepare_memory_hotplug(void)
 {
 #ifdef CONFIG_MEMORY_HOTPLUG
+	preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc");
 	preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap");
 	preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map");
 #endif
-- 
2.39.2