Message-ID: <c38fa123-7750-20de-ed0b-de3346bddc82@quicinc.com>
Date:   Mon, 23 Jan 2023 15:04:20 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.1.1
From:   Sudarshan Rajagopalan <quic_sudaraja@quicinc.com>
Subject: Re: [RFC] memory pressure detection in VMs using PSI mechanism for
 dynamically inflating/deflating VM memory
To:     David Hildenbrand <david@redhat.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Suren Baghdasaryan <surenb@google.com>,
        Mike Rapoport <rppt@kernel.org>,
        Oscar Salvador <osalvador@suse.de>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        <mark.rutland@arm.com>, <will@kernel.org>,
        <virtualization@lists.linux-foundation.org>, <linux-mm@kvack.org>,
        <linux-kernel@vger.kernel.org>,
        <linux-arm-kernel@lists.infradead.org>,
        <linux-arm-msm@vger.kernel.org>
CC:     "Trilok Soni (QUIC)" <quic_tsoni@quicinc.com>,
        "Sukadev Bhattiprolu (QUIC)" <quic_sukadev@quicinc.com>,
        "Srivatsa Vaddagiri (QUIC)" <quic_svaddagi@quicinc.com>,
        "Patrick Daly (QUIC)" <quic_pdaly@quicinc.com>
References: <072de3f4-6bd3-f9ce-024d-e469288fc46a@quicinc.com>
 <acd8d55c-8528-bb21-488e-e5999c3c4e4e@redhat.com>
 <2faf67fe-b1df-d110-6d57-67f284cd5584@quicinc.com>
 <de3f1fd1-bcec-c6e5-e57a-b06674144c3b@redhat.com>
Content-Language: en-US
In-Reply-To: <de3f1fd1-bcec-c6e5-e57a-b06674144c3b@redhat.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 1/23/2023 1:58 AM, David Hildenbrand wrote:
>>>>
>>>> 1. This will be a native userspace daemon that will be running only in
>>>> the Linux VM which will use virtio-mem driver that uses memory hotplug
>>>> to add/remove memory. The VM (aka Secondary VM, SVM) will request for
>>>> memory from the host which is Primary VM, PVM via the backend 
>>>> hypervisor
>>>> which takes care of cross-VM communication.
>>>>
>>>> 2. This will be guest driver. This daemon will use PSI mechanism to
>>>> monitor memory pressure to keep track of memory demands in the system.
>>>> It will register to few memory pressure events and make an educated
>>>> guess on when demand for memory in system is increasing.
>>>
>>> Is that running in the primary or the secondary VM?
>>
>> The userspace PSI daemon will be running on secondary VM. It will talk
>> to a kernel driver (running on secondary VM itself) via ioctl. This
>> kernel driver will talk to slightly modified version of virtio-mem
>> driver where it can call the virtio_mem_config_changed(virtiomem_device)
>> function for resizing the secondary VM. So its mainly "guest driven" 
>> now.
>
> Okay, thanks.
>
> [...]
>
>>>>
>>>> This daemon is currently in just Beta stage now and we have basic
>>>> functionality running. We are yet to add more flesh to this scheme to
>>>
>>> Good to hear that the basics are running with virtio-mem (I assume 
>>> :) ).
>>>
>>>> make sure any potential risks or security concerns are taken care as
>>>> well.
>>>
>>> It would be great to draw/explain the architecture in more detail.
>>
>> We will be looking into solving any potential security concerns where
>> hypervisor would restrict few actions of resizing of memory. Right now,
>> we are experimenting to see if PSI mechanism itself can be used for ways
>> of detecting memory pressure in the system and add memory to secondary
>> VM when memory is in need. Taking into account all the latencies
>> involved in the PSI scheme (i.e. time when one does malloc call till
>> when extra memory gets added to SVM system). And wanted to know
>> upstream's opinion on such a scheme using PSI mechanism for detecting
>> memory pressure and resizing SVM accordingly.
>
> One problematic thing is that adding memory to Linux by virtio-mem 
> eventually consumes memory (e.g., the memmap), especially when having 
> to to add a completely new memory block to Linux.
>
Yes we have thought about this issue as well where-in when system is 
heavily on memory pressure, it would require some memory for memmap 
metadata, and also few other places in memory hotplug that it would need 
to alloc_pages for hot-plugging in. I think this path in memory_hotplug 
may be fixed where it doesn't rely on allocating some small portion of 
memory for hotplugging. But, the purpose memory_hotplug itself wasn't 
for plugging memory on system being in memory pressure :).


> So if you're already under severe memory pressure, these allocations 
> to bring up new memory can fail. The question is, if PSI can notify 
> "early" enough such that this barely happens in practice.
>
> There are some possible ways to mitigate:
>
> 1) Always keep spare memory blocks by virtio-mem added to Linux, that
> B B  don't expose any memory yet. Memory from these block can be handed
> B B  over to Linux without additional Linux allocations. Of course, they
> B B  consume metadata, so one might want to limit them.
>
> 2) Implement memmap_on_memory support for virtio-mem. This might help in
> B B  some setups, where the device block size is suitable.
>
> Did you run into that scenario already during your experiments, and 
> how did you deal with that?
>
We are exactly implementing 2) you had mentioned i.e. enabling 
memmap_on_memory support for virtio-mem. This always guarantees that 
free memory is always present for memmap metadata while hotplugging. But 
this required us to increase memory block size to 256MB (from 128MB) for 
alignment requirement of memory hotplug to enable memory_on_memmap, for 
4K page size configuration. Option 1) you mentioned also seems 
interesting - its good to have some spare memory in hand when system is 
heavily in memory pressure so that this memory can be handed over 
immediately on PSI pressure and doesn't have to wait for memory plug-in 
request roundtrip from Primary VM.

Do you think having memmap_on_memory support for virtio-mem is useful to 
have? If so, we can send the patch that supports this in virtio-mem?

Also, we are looking into ways of having memmap_on_memory enabled 
without requiring to increase memory block size. This might require some 
core changes in memory_hotplug but we haven't explored it much.