Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9aa7d030-19b5-01df-70c0-86d8d6ab86a6@redhat.com>
Date:   Wed, 17 May 2023 15:49:27 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Subject: Re: [PATCH 0/7] riscv: Memory Hot(Un)Plug support
Content-Language: en-US
To:     =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= <bjorn@kernel.org>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        linux-riscv@lists.infradead.org
Cc:     =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= <bjorn@rivosinc.com>,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        Oscar Salvador <osalvador@suse.de>,
        virtualization@lists.linux-foundation.org, linux@rivosinc.com,
        Alexandre Ghiti <alexghiti@rivosinc.com>
References: <20230512145737.985671-1-bjorn@kernel.org>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 12.05.23 16:57, Björn Töpel wrote:
> From: Björn Töpel <bjorn@rivosinc.com>
> 
> Memory Hot(Un)Plug support for the RISC-V port
> ==============================================
> 
> Introduction
> ------------
> 
> To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory
> hot(un)plug allows for increasing and decreasing the size of physical
> memory available to a machine at runtime."
> 
> This series attempts to add memory hot(un)plug support for the RISC-V
> Linux port.
> 
> I'm sending the series as a v1, but it's borderline RFC. It definitely
> needs more testing time, but it would be nice with some early input.
> 
> Implementation
> --------------
> 
>  From an arch perspective, a couple of callbacks needs to be
> implemented to support hot plugging:
> 
> arch_add_memory()
> This callback is responsible for updating the linear/direct map, and
> call into the memory hot plugging generic code via __add_pages().
> 
> arch_remove_memory()
> In this callback the linear/direct map is tore down.
> 
> vmemmap_free()
> The function tears down the vmemmap mappings (if
> CONFIG_SPARSEMEM_VMEMMAP is in-use), and also deallocates the backing
> vmemmap pages. Note that for persistent memory, an alternative
> allocator for the backing pages can be used -- the vmem_altmap. This
> means that when the backing pages are cleared, extra care is needed so
> that the correct deallocation method is used. Note that RISC-V
> populates the vmemmap using vmemmap_populate_basepages(), so currently
> no hugepages are used for the backing store.
> 
> The page table unmap/teardown functions are heavily based (copied!)
> from the x86 tree. The same remove_pgd_mapping() is used in both
> vmemmap_free() and arch_remove_memory(), but in the latter function
> the backing pages are not removed.
> 
> On RISC-V, the PGD level kernel mappings needs to synchronized with
> all page-tables (e.g. via sync_kernel_mappings()). Synchronization
> involves special care, like locking. Instead, this patch series takes
> a different approach (introduced by Jörg Rödel in the x86-tree);
> Pre-allocate the PGD-leaves (P4D, PUD, or PMD depending on the paging
> setup) at mem_init(), for vmemmap and the direct map.
> 
> Pre-allocating the PGD-leaves waste some memory, but is only enabled
> for CONFIG_MEMORY_HOTPLUG. The number pages, potentially unused, are
> ~128 * 4K.
> 
> Patch 1: Preparation for hotplugging support, by pre-allocating the
>           PGD leaves.
> 
> Patch 2: Changes the __init attribute to __meminit, to avoid that the
>           functions are removed after init. __meminit keeps the
>           functions after init, if memory hotplugging is enabled for
>           the build.
>           
> Patch 3: Refactor the direct map setup, so it can be used for hot add.
> 
> Patch 4: The actual add/remove code. Mostly a page-table-walk
>           exercise.
> 
> Patch 5: Turn on the arch support in Kconfig
> 
> Patch 6: Now that memory hotplugging is enabled, make virtio-mem
>           usable for RISC-V
>           
> Patch 7: Pre-allocate vmalloc PGD-leaves as well, which removes the
>           need for vmalloc faulting.
>           
> RFC
> ---
> 
>   * TLB flushes. The current series uses Big Hammer flush-it-all.
>   * Pre-allocation vs explicit syncs
> 
> Testing
> -------
> 
> ACPI support is still in the making for RISC-V, so tests that involve
> CXL and similar fanciness is currently not possible. Virtio-mem,
> however, works without proper ACPI support. In order to try this out
> in Qemu, some additional patches for Qemu are needed:
> 
>   * Enable virtio-mem for RISC-V
>   * Add proper hotplug support for virtio-mem
>   
> The patch for Qemu can be found is commit 5d90a7ef1bc0
> ("hw/riscv/virt: Support for virtio-mem-pci"), and can be found here
> 
>    https://github.com/bjoto/qemu/tree/riscv-virtio-mem
> 
> I will try to upstream that work in parallel with this.
>    
> Thanks to David Hildenbrand for valuable input for the Qemu side of
> things.
> 
> The series is based on the RISC-V fixes tree
>    https://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git/log/?h=fixes
> 

Cool stuff! I'm fairly busy right now, so some high-level questions upfront:

What is the memory section size (which implies the memory block size 
and)? This implies the minimum DIMM granularity and the high-level 
granularity in which virtio-mem adds memory.

What is the pageblock size, implying the minimum granularity that 
virtio-mem can operate on?

On x86-64 and arm64 we currently use the ACPI SRAT to expose the maximum 
physical address where we can see memory getting hotplugged. [1] From 
that, we can derive the "max_possible_pfn" and prepare the kernel 
virtual memory layourt (especially, direct map).

Is something similar required on RISC-V? On s390x, I'm planning on 
adding a paravirtualized mechanism to detect where memory devices might 
be located. (I had a running RFC, but was distracted by all other kinds 
of stuff)


[1] https://virtio-mem.gitlab.io/developer-guide.html

-- 
Thanks,

David / dhildenb