From: Björn Töpel <[email protected]>
Memory Hot(Un)Plug support for the RISC-V port
==============================================
Introduction
------------
To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory
hot(un)plug allows for increasing and decreasing the size of physical
memory available to a machine at runtime."
This series attempts to add memory hot(un)plug support for the RISC-V
Linux port.
I'm sending the series as a v1, but it's borderline RFC. It definitely
needs more testing time, but it would be nice with some early input.
Implementation
--------------
From an arch perspective, a couple of callbacks needs to be
implemented to support hot plugging:
arch_add_memory()
This callback is responsible for updating the linear/direct map, and
call into the memory hot plugging generic code via __add_pages().
arch_remove_memory()
In this callback the linear/direct map is tore down.
vmemmap_free()
The function tears down the vmemmap mappings (if
CONFIG_SPARSEMEM_VMEMMAP is in-use), and also deallocates the backing
vmemmap pages. Note that for persistent memory, an alternative
allocator for the backing pages can be used -- the vmem_altmap. This
means that when the backing pages are cleared, extra care is needed so
that the correct deallocation method is used. Note that RISC-V
populates the vmemmap using vmemmap_populate_basepages(), so currently
no hugepages are used for the backing store.
The page table unmap/teardown functions are heavily based (copied!)
from the x86 tree. The same remove_pgd_mapping() is used in both
vmemmap_free() and arch_remove_memory(), but in the latter function
the backing pages are not removed.
On RISC-V, the PGD level kernel mappings needs to synchronized with
all page-tables (e.g. via sync_kernel_mappings()). Synchronization
involves special care, like locking. Instead, this patch series takes
a different approach (introduced by Jörg Rödel in the x86-tree);
Pre-allocate the PGD-leaves (P4D, PUD, or PMD depending on the paging
setup) at mem_init(), for vmemmap and the direct map.
Pre-allocating the PGD-leaves waste some memory, but is only enabled
for CONFIG_MEMORY_HOTPLUG. The number pages, potentially unused, are
~128 * 4K.
Patch 1: Preparation for hotplugging support, by pre-allocating the
PGD leaves.
Patch 2: Changes the __init attribute to __meminit, to avoid that the
functions are removed after init. __meminit keeps the
functions after init, if memory hotplugging is enabled for
the build.
Patch 3: Refactor the direct map setup, so it can be used for hot add.
Patch 4: The actual add/remove code. Mostly a page-table-walk
exercise.
Patch 5: Turn on the arch support in Kconfig
Patch 6: Now that memory hotplugging is enabled, make virtio-mem
usable for RISC-V
Patch 7: Pre-allocate vmalloc PGD-leaves as well, which removes the
need for vmalloc faulting.
RFC
---
* TLB flushes. The current series uses Big Hammer flush-it-all.
* Pre-allocation vs explicit syncs
Testing
-------
ACPI support is still in the making for RISC-V, so tests that involve
CXL and similar fanciness is currently not possible. Virtio-mem,
however, works without proper ACPI support. In order to try this out
in Qemu, some additional patches for Qemu are needed:
* Enable virtio-mem for RISC-V
* Add proper hotplug support for virtio-mem
The patch for Qemu can be found is commit 5d90a7ef1bc0
("hw/riscv/virt: Support for virtio-mem-pci"), and can be found here
https://github.com/bjoto/qemu/tree/riscv-virtio-mem
I will try to upstream that work in parallel with this.
Thanks to David Hildenbrand for valuable input for the Qemu side of
things.
The series is based on the RISC-V fixes tree
https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/log/?h=fixes
Thanks,
Björn
Björn Töpel (7):
riscv: mm: Pre-allocate PGD leaves to avoid synchronization
riscv: mm: Change attribute from __init to __meminit for page
functions
riscv: mm: Refactor create_linear_mapping_range() for hot add
riscv: mm: Add memory hot add/remove support
riscv: Enable memory hot add/remove arch kbuild support
virtio-mem: Enable virtio-mem for RISC-V
riscv: mm: Pre-allocate vmalloc PGD leaves
arch/riscv/Kconfig | 2 +
arch/riscv/include/asm/kasan.h | 4 +-
arch/riscv/include/asm/mmu.h | 2 +-
arch/riscv/include/asm/pgtable.h | 2 +-
arch/riscv/mm/fault.c | 7 +-
arch/riscv/mm/init.c | 387 ++++++++++++++++++++++++++++---
drivers/virtio/Kconfig | 2 +-
7 files changed, 364 insertions(+), 42 deletions(-)
base-commit: 3b90b09af5be42491a8a74a549318cfa265b3029
--
2.39.2
From: Björn Töpel <[email protected]>
Prepare for memory hot add/remove support by changing from __init to
__meminit for the page-table functions that are used by the upcoming
arch specific callbacks.
Changing the __init attribute to __meminit, avoids that the functions
are removed after init. __meminit keeps the functions after init, if
memory hotplugging is enabled for the build.
Signed-off-by: Björn Töpel <[email protected]>
---
arch/riscv/include/asm/mmu.h | 2 +-
arch/riscv/include/asm/pgtable.h | 2 +-
arch/riscv/mm/init.c | 49 ++++++++++++++------------------
3 files changed, 24 insertions(+), 29 deletions(-)
diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h
index 0099dc116168..9e5d4f37ba2e 100644
--- a/arch/riscv/include/asm/mmu.h
+++ b/arch/riscv/include/asm/mmu.h
@@ -22,7 +22,7 @@ typedef struct {
#endif
} mm_context_t;
-void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa,
+void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa,
phys_addr_t sz, pgprot_t prot);
#endif /* __ASSEMBLY__ */
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 2258b27173b0..a4cdcb689959 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -147,7 +147,7 @@ struct pt_alloc_ops {
#endif
};
-extern struct pt_alloc_ops pt_ops __initdata;
+extern struct pt_alloc_ops pt_ops __meminitdata;
#ifdef CONFIG_MMU
/* Number of PGD entries that a user-mode program can use */
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index d2595cc33a1c..e974ff6ef036 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -356,7 +356,7 @@ static void __init setup_bootmem(void)
}
#ifdef CONFIG_MMU
-struct pt_alloc_ops pt_ops __initdata;
+struct pt_alloc_ops pt_ops __meminitdata;
pgd_t swapper_pg_dir[PTRS_PER_PGD] __page_aligned_bss;
pgd_t trampoline_pg_dir[PTRS_PER_PGD] __page_aligned_bss;
@@ -418,7 +418,7 @@ static inline pte_t *__init get_pte_virt_fixmap(phys_addr_t pa)
return (pte_t *)set_fixmap_offset(FIX_PTE, pa);
}
-static inline pte_t *__init get_pte_virt_late(phys_addr_t pa)
+static inline pte_t *__meminit get_pte_virt_late(phys_addr_t pa)
{
return (pte_t *) __va(pa);
}
@@ -437,7 +437,7 @@ static inline phys_addr_t __init alloc_pte_fixmap(uintptr_t va)
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
}
-static phys_addr_t __init alloc_pte_late(uintptr_t va)
+static phys_addr_t __meminit alloc_pte_late(uintptr_t va)
{
unsigned long vaddr;
@@ -447,9 +447,8 @@ static phys_addr_t __init alloc_pte_late(uintptr_t va)
return __pa(vaddr);
}
-static void __init create_pte_mapping(pte_t *ptep,
- uintptr_t va, phys_addr_t pa,
- phys_addr_t sz, pgprot_t prot)
+static void __meminit create_pte_mapping(pte_t *ptep, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
+ pgprot_t prot)
{
uintptr_t pte_idx = pte_index(va);
@@ -503,7 +502,7 @@ static pmd_t *__init get_pmd_virt_fixmap(phys_addr_t pa)
return (pmd_t *)set_fixmap_offset(FIX_PMD, pa);
}
-static pmd_t *__init get_pmd_virt_late(phys_addr_t pa)
+static pmd_t *__meminit get_pmd_virt_late(phys_addr_t pa)
{
return (pmd_t *) __va(pa);
}
@@ -520,7 +519,7 @@ static phys_addr_t __init alloc_pmd_fixmap(uintptr_t va)
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
}
-static phys_addr_t __init alloc_pmd_late(uintptr_t va)
+static phys_addr_t __meminit alloc_pmd_late(uintptr_t va)
{
unsigned long vaddr;
@@ -530,9 +529,8 @@ static phys_addr_t __init alloc_pmd_late(uintptr_t va)
return __pa(vaddr);
}
-static void __init create_pmd_mapping(pmd_t *pmdp,
- uintptr_t va, phys_addr_t pa,
- phys_addr_t sz, pgprot_t prot)
+static void __meminit create_pmd_mapping(pmd_t *pmdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
+ pgprot_t prot)
{
pte_t *ptep;
phys_addr_t pte_phys;
@@ -568,7 +566,7 @@ static pud_t *__init get_pud_virt_fixmap(phys_addr_t pa)
return (pud_t *)set_fixmap_offset(FIX_PUD, pa);
}
-static pud_t *__init get_pud_virt_late(phys_addr_t pa)
+static pud_t *__meminit get_pud_virt_late(phys_addr_t pa)
{
return (pud_t *)__va(pa);
}
@@ -586,7 +584,7 @@ static phys_addr_t __init alloc_pud_fixmap(uintptr_t va)
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
}
-static phys_addr_t alloc_pud_late(uintptr_t va)
+static phys_addr_t __meminit alloc_pud_late(uintptr_t va)
{
unsigned long vaddr;
@@ -606,7 +604,7 @@ static p4d_t *__init get_p4d_virt_fixmap(phys_addr_t pa)
return (p4d_t *)set_fixmap_offset(FIX_P4D, pa);
}
-static p4d_t *__init get_p4d_virt_late(phys_addr_t pa)
+static p4d_t *__meminit get_p4d_virt_late(phys_addr_t pa)
{
return (p4d_t *)__va(pa);
}
@@ -624,7 +622,7 @@ static phys_addr_t __init alloc_p4d_fixmap(uintptr_t va)
return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
}
-static phys_addr_t alloc_p4d_late(uintptr_t va)
+static phys_addr_t __meminit alloc_p4d_late(uintptr_t va)
{
unsigned long vaddr;
@@ -633,9 +631,8 @@ static phys_addr_t alloc_p4d_late(uintptr_t va)
return __pa(vaddr);
}
-static void __init create_pud_mapping(pud_t *pudp,
- uintptr_t va, phys_addr_t pa,
- phys_addr_t sz, pgprot_t prot)
+static void __meminit create_pud_mapping(pud_t *pudp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
+ pgprot_t prot)
{
pmd_t *nextp;
phys_addr_t next_phys;
@@ -660,9 +657,8 @@ static void __init create_pud_mapping(pud_t *pudp,
create_pmd_mapping(nextp, va, pa, sz, prot);
}
-static void __init create_p4d_mapping(p4d_t *p4dp,
- uintptr_t va, phys_addr_t pa,
- phys_addr_t sz, pgprot_t prot)
+static void __meminit create_p4d_mapping(p4d_t *p4dp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
+ pgprot_t prot)
{
pud_t *nextp;
phys_addr_t next_phys;
@@ -718,9 +714,8 @@ static void __init create_p4d_mapping(p4d_t *p4dp,
#define create_pmd_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0)
#endif /* __PAGETABLE_PMD_FOLDED */
-void __init create_pgd_mapping(pgd_t *pgdp,
- uintptr_t va, phys_addr_t pa,
- phys_addr_t sz, pgprot_t prot)
+void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
+ pgprot_t prot)
{
pgd_next_t *nextp;
phys_addr_t next_phys;
@@ -745,7 +740,7 @@ void __init create_pgd_mapping(pgd_t *pgdp,
create_pgd_next_mapping(nextp, va, pa, sz, prot);
}
-static uintptr_t __init best_map_size(phys_addr_t base, phys_addr_t size)
+static uintptr_t __meminit best_map_size(phys_addr_t base, phys_addr_t size)
{
if (!(base & (PGDIR_SIZE - 1)) && size >= PGDIR_SIZE)
return PGDIR_SIZE;
@@ -778,7 +773,7 @@ asmlinkage void __init __copy_data(void)
#endif
#ifdef CONFIG_STRICT_KERNEL_RWX
-static __init pgprot_t pgprot_from_va(uintptr_t va)
+static __meminit pgprot_t pgprot_from_va(uintptr_t va)
{
if (is_va_kernel_text(va))
return PAGE_KERNEL_READ_EXEC;
@@ -805,7 +800,7 @@ void mark_rodata_ro(void)
debug_checkwx();
}
#else
-static __init pgprot_t pgprot_from_va(uintptr_t va)
+static __meminit pgprot_t pgprot_from_va(uintptr_t va)
{
if (IS_ENABLED(CONFIG_64BIT) && !is_kernel_mapping(va))
return PAGE_KERNEL;
--
2.39.2
From: Björn Töpel <[email protected]>
Enable ARCH_ENABLE_MEMORY_HOTPLUG and ARCH_ENABLE_MEMORY_HOTREMOVE for
RISC-V.
Signed-off-by: Björn Töpel <[email protected]>
---
arch/riscv/Kconfig | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 348c0fa1fc8c..81b3f188f396 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -14,6 +14,8 @@ config RISCV
def_bool y
select ARCH_DMA_DEFAULT_COHERENT
select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
+ select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM && 64BIT && MMU
+ select ARCH_ENABLE_MEMORY_HOTREMOVE if SPARSEMEM && 64BIT && MMU
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_BINFMT_FLAT
--
2.39.2
From: Björn Töpel <[email protected]>
The RISC-V port copies PGD from init_mm to all userland pages-tables,
which means that when the PGD level of the init_mm table is changed,
other page-tables has to be updated.
One way to avoid synchronizing page-tables is to pre-allocate the
pages that are copied (need to be synchronized). For memory
hotswapping builds, prefer to waste some pages, rather than do
explicit synchronization.
Prepare the RISC-V port for memory add/remove, by getting rid of PGD
synchronization. Pre-allocate vmemmap, and direct map pages. This will
roughly waste ~128 worth of 4K pages.
Note that this is only done for memory hotswap enabled configuration.
Signed-off-by: Björn Töpel <[email protected]>
---
arch/riscv/include/asm/kasan.h | 4 +-
arch/riscv/mm/init.c | 86 ++++++++++++++++++++++++++++++++++
2 files changed, 88 insertions(+), 2 deletions(-)
diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h
index 0b85e363e778..e6a0071bdb56 100644
--- a/arch/riscv/include/asm/kasan.h
+++ b/arch/riscv/include/asm/kasan.h
@@ -6,8 +6,6 @@
#ifndef __ASSEMBLY__
-#ifdef CONFIG_KASAN
-
/*
* The following comment was copied from arm64:
* KASAN_SHADOW_START: beginning of the kernel virtual addresses.
@@ -34,6 +32,8 @@
*/
#define KASAN_SHADOW_START ((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK)
#define KASAN_SHADOW_END MODULES_LOWEST_VADDR
+
+#ifdef CONFIG_KASAN
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
void kasan_init(void);
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index 747e5b1ef02d..d2595cc33a1c 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -31,6 +31,7 @@
#include <asm/io.h>
#include <asm/ptdump.h>
#include <asm/numa.h>
+#include <asm/kasan.h>
#include "../kernel/head.h"
@@ -156,6 +157,90 @@ static void __init print_vm_layout(void)
static void print_vm_layout(void) { }
#endif /* CONFIG_DEBUG_VM */
+#ifdef CONFIG_MEMORY_HOTPLUG
+/*
+ * Pre-allocates page-table pages for a specific area in the kernel
+ * page-table. Only the level which needs to be synchronized between
+ * all page-tables is allocated because the synchronization can be
+ * expensive.
+ */
+static void __init preallocate_pgd_pages_range(unsigned long start, unsigned long end,
+ const char *area)
+{
+ unsigned long addr;
+ const char *lvl;
+
+ for (addr = start; addr < end; addr = ALIGN(addr + 1, PGDIR_SIZE)) {
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ lvl = "p4d";
+ p4d = p4d_alloc(&init_mm, pgd, addr);
+ if (!p4d)
+ goto failed;
+
+ if (pgtable_l5_enabled)
+ continue;
+
+ /*
+ * The goal here is to allocate all possibly required
+ * hardware page tables pointed to by the top hardware
+ * level.
+ *
+ * On 4-level systems, the P4D layer is folded away
+ * and the above code does no preallocation. Below,
+ * go down to the pud _software_ level to ensure the
+ * second hardware level is allocated on 4-level
+ * systems too.
+ */
+ lvl = "pud";
+ pud = pud_alloc(&init_mm, p4d, addr);
+ if (!pud)
+ goto failed;
+
+ if (pgtable_l4_enabled)
+ continue;
+ /*
+ * The goal here is to allocate all possibly required
+ * hardware page tables pointed to by the top hardware
+ * level.
+ *
+ * On 3-level systems, the PUD layer is folded away
+ * and the above code does no preallocation. Below,
+ * go down to the pmd _software_ level to ensure the
+ * second hardware level is allocated on 3-level
+ * systems too.
+ */
+ lvl = "pmd";
+ pmd = pmd_alloc(&init_mm, pud, addr);
+ if (!pmd)
+ goto failed;
+ }
+
+ return;
+
+failed:
+
+ /*
+ * The pages have to be there now or they will be missing in
+ * process page-tables later.
+ */
+ panic("Failed to pre-allocate %s pages for %s area\n", lvl, area);
+}
+
+#define PAGE_END KASAN_SHADOW_START
+#endif
+
+static void __init prepare_memory_hotplug(void)
+{
+#ifdef CONFIG_MEMORY_HOTPLUG
+ preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap");
+ preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map");
+#endif
+}
+
void __init mem_init(void)
{
#ifdef CONFIG_FLATMEM
@@ -164,6 +249,7 @@ void __init mem_init(void)
swiotlb_init(max_pfn > PFN_DOWN(dma32_phys_limit), SWIOTLB_VERBOSE);
memblock_free_all();
+ prepare_memory_hotplug();
print_vm_layout();
}
--
2.39.2
From: Björn Töpel <[email protected]>
Now that RISC-V has memory hot add/remove support, virtio-mem can be
used on the platform.
Signed-off-by: Björn Töpel <[email protected]>
---
drivers/virtio/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 0a53a61231c2..358e79ece169 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -117,7 +117,7 @@ config VIRTIO_BALLOON
config VIRTIO_MEM
tristate "Virtio mem driver"
- depends on X86_64 || ARM64
+ depends on X86_64 || ARM64 || RISCV
depends on VIRTIO
depends on MEMORY_HOTPLUG
depends on MEMORY_HOTREMOVE
--
2.39.2
From: Björn Töpel <[email protected]>
Instead of relying on vmalloc_fault() to synchronize the page-tables,
pre-allocate the PGD leaves of the vmalloc area. This is only enabled
if memory hot/add is enabled by the build.
Signed-off-by: Björn Töpel <[email protected]>
---
arch/riscv/mm/fault.c | 7 ++++++-
arch/riscv/mm/init.c | 1 +
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 8685f85a7474..b61e279acd50 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -233,12 +233,17 @@ void handle_page_fault(struct pt_regs *regs)
* Fault-in kernel-space virtual memory on-demand.
* The 'reference' page table is init_mm.pgd.
*
+ * For memory hotplug enabled systems, the PGD entries are
+ * pre-allocated, which avoids the need to synchronize
+ * pgd/fault-in.
+ *
* NOTE! We MUST NOT take any locks for this case. We may
* be in an interrupt or a critical region, and should
* only copy the information from the master page table,
* nothing more.
*/
- if (unlikely((addr >= VMALLOC_START) && (addr < VMALLOC_END))) {
+ if (unlikely(!IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
+ (addr >= VMALLOC_START) && (addr < VMALLOC_END))) {
vmalloc_fault(regs, code, addr);
return;
}
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index a468708d1e1c..fd5a6d3fe182 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -236,6 +236,7 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon
static void __init prepare_memory_hotplug(void)
{
#ifdef CONFIG_MEMORY_HOTPLUG
+ preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc");
preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap");
preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map");
#endif
--
2.39.2
On 12.05.23 16:57, Björn Töpel wrote:
> From: Björn Töpel <[email protected]>
>
> Memory Hot(Un)Plug support for the RISC-V port
> ==============================================
>
> Introduction
> ------------
>
> To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory
> hot(un)plug allows for increasing and decreasing the size of physical
> memory available to a machine at runtime."
>
> This series attempts to add memory hot(un)plug support for the RISC-V
> Linux port.
>
> I'm sending the series as a v1, but it's borderline RFC. It definitely
> needs more testing time, but it would be nice with some early input.
>
> Implementation
> --------------
>
> From an arch perspective, a couple of callbacks needs to be
> implemented to support hot plugging:
>
> arch_add_memory()
> This callback is responsible for updating the linear/direct map, and
> call into the memory hot plugging generic code via __add_pages().
>
> arch_remove_memory()
> In this callback the linear/direct map is tore down.
>
> vmemmap_free()
> The function tears down the vmemmap mappings (if
> CONFIG_SPARSEMEM_VMEMMAP is in-use), and also deallocates the backing
> vmemmap pages. Note that for persistent memory, an alternative
> allocator for the backing pages can be used -- the vmem_altmap. This
> means that when the backing pages are cleared, extra care is needed so
> that the correct deallocation method is used. Note that RISC-V
> populates the vmemmap using vmemmap_populate_basepages(), so currently
> no hugepages are used for the backing store.
>
> The page table unmap/teardown functions are heavily based (copied!)
> from the x86 tree. The same remove_pgd_mapping() is used in both
> vmemmap_free() and arch_remove_memory(), but in the latter function
> the backing pages are not removed.
>
> On RISC-V, the PGD level kernel mappings needs to synchronized with
> all page-tables (e.g. via sync_kernel_mappings()). Synchronization
> involves special care, like locking. Instead, this patch series takes
> a different approach (introduced by Jörg Rödel in the x86-tree);
> Pre-allocate the PGD-leaves (P4D, PUD, or PMD depending on the paging
> setup) at mem_init(), for vmemmap and the direct map.
>
> Pre-allocating the PGD-leaves waste some memory, but is only enabled
> for CONFIG_MEMORY_HOTPLUG. The number pages, potentially unused, are
> ~128 * 4K.
>
> Patch 1: Preparation for hotplugging support, by pre-allocating the
> PGD leaves.
>
> Patch 2: Changes the __init attribute to __meminit, to avoid that the
> functions are removed after init. __meminit keeps the
> functions after init, if memory hotplugging is enabled for
> the build.
>
> Patch 3: Refactor the direct map setup, so it can be used for hot add.
>
> Patch 4: The actual add/remove code. Mostly a page-table-walk
> exercise.
>
> Patch 5: Turn on the arch support in Kconfig
>
> Patch 6: Now that memory hotplugging is enabled, make virtio-mem
> usable for RISC-V
>
> Patch 7: Pre-allocate vmalloc PGD-leaves as well, which removes the
> need for vmalloc faulting.
>
> RFC
> ---
>
> * TLB flushes. The current series uses Big Hammer flush-it-all.
> * Pre-allocation vs explicit syncs
>
> Testing
> -------
>
> ACPI support is still in the making for RISC-V, so tests that involve
> CXL and similar fanciness is currently not possible. Virtio-mem,
> however, works without proper ACPI support. In order to try this out
> in Qemu, some additional patches for Qemu are needed:
>
> * Enable virtio-mem for RISC-V
> * Add proper hotplug support for virtio-mem
>
> The patch for Qemu can be found is commit 5d90a7ef1bc0
> ("hw/riscv/virt: Support for virtio-mem-pci"), and can be found here
>
> https://github.com/bjoto/qemu/tree/riscv-virtio-mem
>
> I will try to upstream that work in parallel with this.
>
> Thanks to David Hildenbrand for valuable input for the Qemu side of
> things.
>
> The series is based on the RISC-V fixes tree
> https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/log/?h=fixes
>
Cool stuff! I'm fairly busy right now, so some high-level questions upfront:
What is the memory section size (which implies the memory block size
and)? This implies the minimum DIMM granularity and the high-level
granularity in which virtio-mem adds memory.
What is the pageblock size, implying the minimum granularity that
virtio-mem can operate on?
On x86-64 and arm64 we currently use the ACPI SRAT to expose the maximum
physical address where we can see memory getting hotplugged. [1] From
that, we can derive the "max_possible_pfn" and prepare the kernel
virtual memory layourt (especially, direct map).
Is something similar required on RISC-V? On s390x, I'm planning on
adding a paravirtualized mechanism to detect where memory devices might
be located. (I had a running RFC, but was distracted by all other kinds
of stuff)
[1] https://virtio-mem.gitlab.io/developer-guide.html
--
Thanks,
David / dhildenb
David Hildenbrand <[email protected]> writes:
> On 12.05.23 16:57, Björn Töpel wrote:
>> From: Björn Töpel <[email protected]>
>>
>> Memory Hot(Un)Plug support for the RISC-V port
>> ==============================================
[...]
>
> Cool stuff! I'm fairly busy right now, so some high-level questions upfront:
No worries, and no rush! I'd say the v1 series was mainly for the RISC-V
folks, and I've got tons of (offline) comments from Alex -- and with
your comments below some more details to figure out.
> What is the memory section size (which implies the memory block size
> and)? This implies the minimum DIMM granularity and the high-level
> granularity in which virtio-mem adds memory.
It's 128M (27 bits) -- (like arm64 and x86-64?).
> What is the pageblock size, implying the minimum granularity that
> virtio-mem can operate on?
Nothing special AFAIU; MAX_ORDER is 10, so PAGE_SIZE (4K) * 1024. Hmm, I
realize that I need to look into some more details of virtio-mem! :-)
> On x86-64 and arm64 we currently use the ACPI SRAT to expose the maximum
> physical address where we can see memory getting hotplugged. [1] From
> that, we can derive the "max_possible_pfn" and prepare the kernel
> virtual memory layourt (especially, direct map).
>
> Is something similar required on RISC-V?
Yes! RISC-V is in the progress of getting proper ACPI support. Thanks
for pointing me in the these directions; Food for thought that I'll
digest for the next version.
Cheers,
Björn
Hi David and Anshuman!
Björn Töpel <[email protected]> writes:
> David Hildenbrand <[email protected]> writes:
>
>> On 12.05.23 16:57, Björn Töpel wrote:
>>> From: Björn Töpel <[email protected]>
>>>
>>> Memory Hot(Un)Plug support for the RISC-V port
>>> ==============================================
>
> [...]
>
>>
>> Cool stuff! I'm fairly busy right now, so some high-level questions upfront:
>
> No worries, and no rush! I'd say the v1 series was mainly for the RISC-V
> folks, and I've got tons of (offline) comments from Alex -- and with
> your comments below some more details to figure out.
One of the major issues with my v1 patch is around init_mm page table
synchronization, and that'll be part of the v2.
I've noticed there's a quite a difference between x86-64 and arm64 in
terms of locking, when updating (add/remove) the init_mm table. x86-64
uses the usual page table locking mechanisms (used by the generic
kernel functions), whereas arm64 does not.
How does arm64 manage to mix the "lock-less" updates (READ/WRITE_ONCE,
and fences in set_p?d+friends), with the generic kernel ones that uses
the regular page locking mechanism?
I'm obviously missing something about the locking rules for memory hot
add/remove... I've been reading the arm64 memory hot add/remove
series, but none the wiser! ;-)
Björn
On 21.05.23 11:15, Björn Töpel wrote:
> Hi David and Anshuman!
>
> Björn Töpel <[email protected]> writes:
>
>> David Hildenbrand <[email protected]> writes:
>>
>>> On 12.05.23 16:57, Björn Töpel wrote:
>>>> From: Björn Töpel <[email protected]>
>>>>
>>>> Memory Hot(Un)Plug support for the RISC-V port
>>>> ==============================================
>>
>> [...]
>>
>>>
>>> Cool stuff! I'm fairly busy right now, so some high-level questions upfront:
>>
>> No worries, and no rush! I'd say the v1 series was mainly for the RISC-V
>> folks, and I've got tons of (offline) comments from Alex -- and with
>> your comments below some more details to figure out.
>
> One of the major issues with my v1 patch is around init_mm page table
> synchronization, and that'll be part of the v2.
>
> I've noticed there's a quite a difference between x86-64 and arm64 in
> terms of locking, when updating (add/remove) the init_mm table. x86-64
> uses the usual page table locking mechanisms (used by the generic
> kernel functions), whereas arm64 does not.
>
> How does arm64 manage to mix the "lock-less" updates (READ/WRITE_ONCE,
> and fences in set_p?d+friends), with the generic kernel ones that uses
> the regular page locking mechanism?
>
> I'm obviously missing something about the locking rules for memory hot
> add/remove... I've been reading the arm64 memory hot add/remove
> series, but none the wiser! ;-)
In general, memory hot(un)plug is serialized on a high level using the
mem_hotplug_lock. For example, in pagemap_range() or in
add_memory_resource(), we grab that lock in write mode. So we'll never
see memory getting added/removed concurrently from the direct map.
From what I recall, the locking on the arch level is required for
concurrent (direct mapping) page table modifications that target virtual
address ranges adjacent to the ranges we hot(un)plug:
CONFIG_ARCH_HAS_SET_DIRECT_MAP and vmalloc come to mind.
For example, if a range would be mapped using a large PUD, but we have
to unplug it partially (unplugging memory part of bootmem), we'd have to
replace the large PUD by a PMD table first. That change (that could
affect other concurrent page table walkers/operations) has to be
synchronized.
I guess to which degree this applies to riscv depends on the virtual
memory layout, direct mapping granularity and features (e.g.,
CONFIG_ARCH_HAS_SET_DIRECT_MAP).
One trick that arm64 implements is, that it only allows hotunplugging
memory that was hotplugged (see prevent_bootmem_remove_notifier()). That
might just rule out such problematic cases that require locking
completely, and the high-level mem_hotplug_lock sufficient.
--
Thanks,
David / dhildenb