For now, SEV pins guest's memory to avoid swapping or
moving ciphertext, but leading to the inhibition of
Memory Ballooning.
In Memory Ballooning, only guest's free pages will be relocated
in balloon inflation and deflation, so the difference of plaintext
doesn't matter to guest.
Memory Ballooning is a nice memory overcommitment technology can
be used in CVM based on SEV and SEV-ES, so userspace tools can
provide an option to allow SEV not to pin memory and enable
Memory Ballooning. Guest kernel may not inhibit Balloon and
should set shared memory for Balloon decrypted.
Signed-off-by: Zheyun Shen <[email protected]>
---
drivers/virtio/virtio_balloon.c | 18 ++++++++++++++++++
drivers/virtio/virtio_ring.c | 7 +++++++
2 files changed, 25 insertions(+)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 1fe93e93f..aca4c8a58 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -18,6 +18,9 @@
#include <linux/wait.h>
#include <linux/mm.h>
#include <linux/page_reporting.h>
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+#include <linux/set_memory.h>
+#endif
/*
* Balloon device works in 4K page units. So each page is pointed to by
@@ -870,6 +873,9 @@ static int virtio_balloon_register_shrinker(struct virtio_balloon *vb)
static int virtballoon_probe(struct virtio_device *vdev)
{
struct virtio_balloon *vb;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ size_t vb_size = PAGE_ALIGN(sizeof(*vb));
+#endif
int err;
if (!vdev->config->get) {
@@ -878,11 +884,19 @@ static int virtballoon_probe(struct virtio_device *vdev)
return -EINVAL;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ vdev->priv = vb = kzalloc(vb_size, GFP_KERNEL);
+#else
vdev->priv = vb = kzalloc(sizeof(*vb), GFP_KERNEL);
+#endif
if (!vb) {
err = -ENOMEM;
goto out;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ set_memory_decrypted((unsigned long)vb, vb_size / PAGE_SIZE);
+ memset(vb, 0, vb_size);
+#endif
INIT_WORK(&vb->update_balloon_stats_work, update_balloon_stats_func);
INIT_WORK(&vb->update_balloon_size_work, update_balloon_size_func);
@@ -1101,7 +1115,11 @@ static int virtballoon_validate(struct virtio_device *vdev)
else if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON))
__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_REPORTING);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ __virtio_set_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
+#else
__virtio_clear_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
+#endif
return 0;
}
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 49299b1f9..875612a2e 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -14,6 +14,9 @@
#include <linux/kmsan.h>
#include <linux/spinlock.h>
#include <xen/xen.h>
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+#include <linux/set_memory.h>
+#endif
#ifdef DEBUG
/* For development, we want to crash whenever the ring is screwed. */
@@ -321,6 +324,10 @@ static void *vring_alloc_queue(struct virtio_device *vdev, size_t size,
if (queue) {
phys_addr_t phys_addr = virt_to_phys(queue);
*dma_handle = (dma_addr_t)phys_addr;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ set_memory_decrypted((unsigned long)queue, PAGE_ALIGN(size) / PAGE_SIZE);
+ memset(queue, 0, PAGE_ALIGN(size));
+#endif
/*
* Sanity check: make sure we dind't truncate
--
2.34.1
On Wed, Jan 10, 2024 at 02:22:42PM +0800, Zheyun Shen wrote:
> For now, SEV pins guest's memory to avoid swapping or
> moving ciphertext, but leading to the inhibition of
> Memory Ballooning.
>
> In Memory Ballooning, only guest's free pages will be relocated
> in balloon inflation and deflation, so the difference of plaintext
> doesn't matter to guest.
>
> Memory Ballooning is a nice memory overcommitment technology can
> be used in CVM based on SEV and SEV-ES, so userspace tools can
> provide an option to allow SEV not to pin memory and enable
> Memory Ballooning. Guest kernel may not inhibit Balloon and
> should set shared memory for Balloon decrypted.
>
> Signed-off-by: Zheyun Shen <[email protected]>
Sorry I don't get what you are saying at all.
Please format the commit log along the following lines:
Currently .....
This is bad because ...
To fix ...
As a result ...
> ---
> drivers/virtio/virtio_balloon.c | 18 ++++++++++++++++++
> drivers/virtio/virtio_ring.c | 7 +++++++
> 2 files changed, 25 insertions(+)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 1fe93e93f..aca4c8a58 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -18,6 +18,9 @@
> #include <linux/wait.h>
> #include <linux/mm.h>
> #include <linux/page_reporting.h>
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> +#include <linux/set_memory.h>
> +#endif
>
> /*
> * Balloon device works in 4K page units. So each page is pointed to by
> @@ -870,6 +873,9 @@ static int virtio_balloon_register_shrinker(struct virtio_balloon *vb)
> static int virtballoon_probe(struct virtio_device *vdev)
> {
> struct virtio_balloon *vb;
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + size_t vb_size = PAGE_ALIGN(sizeof(*vb));
> +#endif
> int err;
>
> if (!vdev->config->get) {
> @@ -878,11 +884,19 @@ static int virtballoon_probe(struct virtio_device *vdev)
> return -EINVAL;
> }
>
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + vdev->priv = vb = kzalloc(vb_size, GFP_KERNEL);
> +#else
> vdev->priv = vb = kzalloc(sizeof(*vb), GFP_KERNEL);
> +#endif
> if (!vb) {
> err = -ENOMEM;
> goto out;
> }
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + set_memory_decrypted((unsigned long)vb, vb_size / PAGE_SIZE);
> + memset(vb, 0, vb_size);
> +#endif
>
> INIT_WORK(&vb->update_balloon_stats_work, update_balloon_stats_func);
> INIT_WORK(&vb->update_balloon_size_work, update_balloon_size_func);
> @@ -1101,7 +1115,11 @@ static int virtballoon_validate(struct virtio_device *vdev)
> else if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON))
> __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_REPORTING);
>
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + __virtio_set_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
> +#else
> __virtio_clear_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
> +#endif
> return 0;
> }
>
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index 49299b1f9..875612a2e 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -14,6 +14,9 @@
> #include <linux/kmsan.h>
> #include <linux/spinlock.h>
> #include <xen/xen.h>
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> +#include <linux/set_memory.h>
> +#endif
>
> #ifdef DEBUG
> /* For development, we want to crash whenever the ring is screwed. */
> @@ -321,6 +324,10 @@ static void *vring_alloc_queue(struct virtio_device *vdev, size_t size,
> if (queue) {
> phys_addr_t phys_addr = virt_to_phys(queue);
> *dma_handle = (dma_addr_t)phys_addr;
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + set_memory_decrypted((unsigned long)queue, PAGE_ALIGN(size) / PAGE_SIZE);
> + memset(queue, 0, PAGE_ALIGN(size));
> +#endif
>
> /*
> * Sanity check: make sure we dind't truncate
No way I am going to spead CONFIG_AMD_MEM_ENCRYPT all over the place
like this.
> --
> 2.34.1
On Wed, Jan 10, 2024 at 2:23 PM Zheyun Shen <[email protected]> wrote:
>
> For now, SEV pins guest's memory to avoid swapping or
> moving ciphertext, but leading to the inhibition of
> Memory Ballooning.
>
> In Memory Ballooning, only guest's free pages will be relocated
> in balloon inflation and deflation, so the difference of plaintext
> doesn't matter to guest.
This seems only true if the page is zeroed, is this true here?
Thanks
On Wed, Jan 10, 2024 at 4:01 PM Michael S. Tsirkin <[email protected]> wrote:
> Sorry I don't get what you are saying at all.
> Please format the commit log along the following lines:
> Currently .....
> This is bad because ...
> To fix ...
> As a result ...
> No way I am going to spead CONFIG_AMD_MEM_ENCRYPT all over the place
> like this.
I will try to find out a solution with fewer macros and send patch V2
with a more perspicuous commit log.
On Thur, Jan 11, 2024 at 11:20 AM Jason Wang <[email protected]> wrote:
> > For now, SEV pins guest's memory to avoid swapping or
> > moving ciphertext, but leading to the inhibition of
> > Memory Ballooning.
> >
> > In Memory Ballooning, only guest's free pages will be relocated
> > in balloon inflation and deflation, so the difference of plaintext
> > doesn't matter to guest.
> This seems only true if the page is zeroed, is this true here?
Sorry, I cannot figure out why the pages should be zeroed. I think
both host kernel and guest kernel assume that the pages are not
zeroed and will use kzalloc or manually zero them in real applications,
which is same as non-SEV environments.
I have tested in SEV-ES, reclaiming memory by balloon inflation and reuse
them after balloon deflation both works well with the patch. Hypervisor
can normally give the reclaimed memory from one CVM to another, or give
back to the origin CVM.
>>> For now, SEV pins guest's memory to avoid swapping or
>>> moving ciphertext, but leading to the inhibition of
>>> Memory Ballooning.
>>>
>>> In Memory Ballooning, only guest's free pages will be relocated
>>> in balloon inflation and deflation, so the difference of plaintext
>>> doesn't matter to guest.
>
>> This seems only true if the page is zeroed, is this true here?
>
> Sorry, I cannot figure out why the pages should be zeroed. I think
> both host kernel and guest kernel assume that the pages are not
> zeroed and will use kzalloc or manually zero them in real applications,
> which is same as non-SEV environments.
balloon_page_alloc() will not zero the memory (no __GFP_ZERO set). Only
in some configurations (zero-on-alloc, zero-on-free), the kernel would
do that implicitly.
So we'd eventually be leaking secrets to the untrusted hypervisor?
> I have tested in SEV-ES, reclaiming memory by balloon inflation and reuse
> them after balloon deflation both works well with the patch. Hypervisor
> can normally give the reclaimed memory from one CVM to another, or give
> back to the origin CVM.
I'll comment on your misconception of memory overcommit separately.
--
Cheers,
David / dhildenb
On 10.01.24 07:22, Zheyun Shen wrote:
> For now, SEV pins guest's memory to avoid swapping or
> moving ciphertext, but leading to the inhibition of
> Memory Ballooning.
>
> In Memory Ballooning, only guest's free pages will be relocated
> in balloon inflation and deflation, so the difference of plaintext
> doesn't matter to guest.
A Linux hypervisor will always give you a fresh, zeroed page. I don't
recall what the spec says, could be that that is a guarantee.
>
> Memory Ballooning is a nice memory overcommitment technology can
> be used in CVM based on SEV and SEV-ES, so userspace tools can
> provide an option to allow SEV not to pin memory and enable
> Memory Ballooning. Guest kernel may not inhibit Balloon and
> should set shared memory for Balloon decrypted.
Two points:
1) Memory overcommit means that you promise to have more memory than you
actually have.
To be able to use that in a *safe* way in the hypervisor, to fulfill
that promise, you need some backup strategy, which is usually swap space
in the hypervisor. Further one might apply other techniques (ram
compression, memory deduplication) in the hypervisor to make that
swapping unlikely to ever happen when overcommitting (because nobody
wants to swap).
Assume you run a lot of VMs that mostly have private/encrypted memory
(which is the default). Imagine you previously inflated the balloon on
VM0, and that VM needs more memory (you promised it could have more!).
You reach out to other VMs to inflate the balloon so you get memory
back, but they cannot give up memory safely.
In that scenario (a) you cannot swap something out because all pages are
pinned (b) memory compression cannot be applied because pages are pinned
and (c) memory deduplication cannot be applied because pages are pinned.
Pinned memory is a resource that cannot be overcomitted.
So I am not convinced the use case you are targeting can be considered
any way of sane memory overcommit. You should better call it resizing VM
memory instead. Then, it's clearer that the hypervisor cannot promise to
ever give you that memory when you are in need.
2) What about other features?
What if the hypervisor enabled free-page-reporting? Would that work
(unlikely, I assunme). Don't we have to block that?
--
Cheers,
David / dhildenb