Message-ID: <1bf30145-22a5-cc46-e583-25053460b105@redhat.com>
Date:   Tue, 24 Jan 2023 16:20:31 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.0
Content-Language: en-US
To:     Sudarshan Rajagopalan <quic_sudaraja@quicinc.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Suren Baghdasaryan <surenb@google.com>,
        Mike Rapoport <rppt@kernel.org>,
        Oscar Salvador <osalvador@suse.de>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        mark.rutland@arm.com, will@kernel.org,
        virtualization@lists.linux-foundation.org, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
        linux-arm-msm@vger.kernel.org
Cc:     "Trilok Soni (QUIC)" <quic_tsoni@quicinc.com>,
        "Sukadev Bhattiprolu (QUIC)" <quic_sukadev@quicinc.com>,
        "Srivatsa Vaddagiri (QUIC)" <quic_svaddagi@quicinc.com>,
        "Patrick Daly (QUIC)" <quic_pdaly@quicinc.com>
References: <072de3f4-6bd3-f9ce-024d-e469288fc46a@quicinc.com>
 <acd8d55c-8528-bb21-488e-e5999c3c4e4e@redhat.com>
 <2faf67fe-b1df-d110-6d57-67f284cd5584@quicinc.com>
 <de3f1fd1-bcec-c6e5-e57a-b06674144c3b@redhat.com>
 <c38fa123-7750-20de-ed0b-de3346bddc82@quicinc.com>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
Subject: Re: [RFC] memory pressure detection in VMs using PSI mechanism for
 dynamically inflating/deflating VM memory
In-Reply-To: <c38fa123-7750-20de-ed0b-de3346bddc82@quicinc.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 24.01.23 00:04, Sudarshan Rajagopalan wrote:
[...]
>> One problematic thing is that adding memory to Linux by virtio-mem
>> eventually consumes memory (e.g., the memmap), especially when having
>> to to add a completely new memory block to Linux.
>>
> Yes we have thought about this issue as well where-in when system is
> heavily on memory pressure, it would require some memory for memmap
> metadata, and also few other places in memory hotplug that it would need
> to alloc_pages for hot-plugging in. I think this path in memory_hotplug
> may be fixed where it doesn't rely on allocating some small portion of
> memory for hotplugging. But, the purpose memory_hotplug itself wasn't
> for plugging memory on system being in memory pressure :).

Some small allocations might be classified as "urgent" and go to atomic 
reserves (e.g., resource tree node, memory device node). The big 
allocations (memmap, page-ext if enabled, eventually page tables for 
direct map when not mapping huge pages) are the problematic "memory 
consumers" I think.

> 
> 
>> So if you're already under severe memory pressure, these allocations
>> to bring up new memory can fail. The question is, if PSI can notify
>> "early" enough such that this barely happens in practice.
>>
>> There are some possible ways to mitigate:
>>
>> 1) Always keep spare memory blocks by virtio-mem added to Linux, that
>> B B  don't expose any memory yet. Memory from these block can be handed
>> B B  over to Linux without additional Linux allocations. Of course, they
>> B B  consume metadata, so one might want to limit them.
>>
>> 2) Implement memmap_on_memory support for virtio-mem. This might help in
>> B B  some setups, where the device block size is suitable.
>>
>> Did you run into that scenario already during your experiments, and
>> how did you deal with that?
>>
> We are exactly implementing 2) you had mentioned i.e. enabling
> memmap_on_memory support for virtio-mem. This always guarantees that
> free memory is always present for memmap metadata while hotplugging. But
> this required us to increase memory block size to 256MB (from 128MB) for
> alignment requirement of memory hotplug to enable memory_on_memmap, for
> 4K page size configuration. Option 1) you mentioned also seems

The memmap of 128 MiB is 2 MiB. Assuming the pageblock size is 2 MiB, 
and virtio-mem supports a device block size of 2 MiB, it should "in 
theory" also work with 128 MiB memory blocks.

So I'd be curious why the change to 256 MiB was required. Maybe, that 
kernel config ends up with a pageblock size of 4 MiB (IIRC that can 
happen without CONFIG_HUGETLB -- which we should most probbaly change to 
also be PMD_ORDER due to THP).

> interesting - its good to have some spare memory in hand when system is
> heavily in memory pressure so that this memory can be handed over
> immediately on PSI pressure and doesn't have to wait for memory plug-in
> request roundtrip from Primary VM.

The idea was that you'd still do the roundtrip to request plugging of 
device memory blocks, but that you could immediately expose memory to 
the system (without requiring allocations), to eventually immediately 
prepare the next Linux memory block while "fresh" memory is available.

This way you could handle most allocations that happen when adding a 
Linux memory block.

The main idea was to always have at least one spare one lying around. 
And as soon as you start exposing memory from one of them to the page 
allocator, immediately prepare the next one.

> 
> Do you think having memmap_on_memory support for virtio-mem is useful to
> have? If so, we can send the patch that supports this in virtio-mem?
> 

I think yes. However, last time I though about adding support, I 
realized that there are some ugly corner cases to handle cleanly.

You have to make sure that the device memory blocks to-be-used as memmap 
are "plugged" even before calling add_memory_driver_managed(). And you 
can only "unplug" these device memory blocks after the memory block was 
removed via offline_and_remove_memory().

So the whole order of events and management of plugged device blocks 
changes quite a bit ...

... and what to do if the device block size is, say 4MiB, but the memmap 
is 2 MiB? Of course, one could simply skip the optimization then.

Having that said, if you managed to get it running and it's not too 
ugly, please share.

-- 
Thanks,

David / dhildenb