Hi all!
We are planning on submitting a CFP to host a KVM Microconference at
Linux Plumbers Conference 2023. To help justify the proposal, we would
like to gather a list of folks that would likely attend, and crowdsource
a list of topics to include in the proposal.
For both this year and future years, the intent is that a KVM
Microconference will complement KVM Forum, *NOT* supplant it. As you
probably noticed, KVM Forum is going through a somewhat radical change in
how it's organized; the conference is now free and (with some help from
Red Hat) organized directly by the KVM and QEMU communities. Despite the
unexpected changes and some teething pains, community response to KVM
Forum continues to be overwhelmingly positive! KVM Forum will remain
the venue of choice for KVM/userspace collaboration, for educational
content covering both KVM and userspace, and to discuss new features in
QEMU and other userspace projects.
At least on the x86 side, however, the success of KVM Forum led us
virtualization folks to operate in relative isolation. KVM depends on
and impacts multiple subsystems (MM, scheduler, perf) in profound ways,
and recently we’ve seen more and more ideas/features that require
non-trivial changes outside KVM and buy-in from stakeholders that
(typically) do not attend KVM Forum. Linux Plumbers Conference is a
natural place to establish such collaboration within the kernel.
Therefore, the aim of the KVM Microconference will be:
* to provide a setting in which to discuss KVM and kernel internals
* to increase collaboration and reduce friction with other subsystems
* to discuss system virtualization issues that require coordination with
other subsystems (such as VFIO, or guest support in arch/)
Below is a rough draft of the planned CFP submission.
Thanks!
Paolo Bonzini (KVM Maintainer)
Sean Christopherson (KVM x86 Co-Maintainer)
Marc Zyngier (KVM ARM Co-Maintainer)
===================
KVM Microconference
===================
KVM (Kernel-based Virtual Machine) enables the use of hardware features
to improve the efficiency, performance, and security of virtual machines
created and managed by userspace. KVM was originally developed to host
and accelerate "full" virtual machines running a traditional kernel and
operating system, but has long since expanded to cover a wide array of use
cases, e.g. hosting real time workloads, sandboxing untrusted workloads,
deprivileging third party code, reducing the trusted computed base of
security sensitive workloads, etc. As KVM's use cases have grown, so too
have the requirements placed on KVM and the interactions between it and
other kernel subsystems.
The KVM Microconference will focus on how to evolve KVM and adjacent
subsystems in order to satisfy new and upcoming requirements: serving
guest memory that cannot be accessed by host userspace[1], providing
accurate, feature-rich PMU/perf virtualization in cloud VMs[2], etc.
Potential Topics:
- Serving inaccessible/unmappable memory for KVM guests (protected VMs)
- Optimizing mmu_notifiers, e.g. reducing TLB flushes and spurious zapping
- Supporting multiple KVM modules (for non-disruptive upgrades)
- Improving and hardening KVM+perf interactions
- Implementing arch-agnostic abstractions in KVM (e.g. MMU)
- Defining KVM requirements for hardware vendors
- Utilizing "fault" injection to increase test coverage of edge cases
- KVM vs VFIO (e.g. memory types, a rather hot topic on the ARM side)
Key Attendees:
- Paolo Bonzini <[email protected]> (KVM Maintainer)
- Sean Christopherson <[email protected]> (KVM x86 Co-Maintainer)
- Your name could be here!
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/CALMp9eRBOmwz=mspp0m5Q093K3rMUeAsF3vEL39MGV5Br9wEQQ@mail.gmail.com
On Tue, 2023-05-09 at 11:55 +0200, Paolo Bonzini wrote:
> Hi all!
>
> We are planning on submitting a CFP to host a KVM Microconference at
> Linux Plumbers Conference 2023. To help justify the proposal, we would
> like to gather a list of folks that would likely attend, and crowdsource
> a list of topics to include in the proposal.
Hi Paolo,
This MC sounds great! There are two topics I'd be keen to discuss, both in
the KVM + memory-management realm:
1. Guest and kernel memory persistence across kexec for live update.
Specifically focussing on the host IOMMU pgtable persistence for DMA-
passthrough devices to support kexec while guest-driven DMA is still
running. There is some discussion happening now about this [1] and
hopefully the discussion and prototyping will continue in the run up to
LPC.
2. Supporting more fine-grain memory management and access control APIs
for the virtualisation case specifically, for use-cases around live
migration, memory oversubscription, and "side-car" virtual machines. These
use cases would benefit from kernel support for things like dynamically
updating IOMMU and MMU permissions independently at fine granularity, all
without actually modifying the VMAs, to support fine-grain handling. And
linking this topic to the one above: being able to do these things when
not backed by struct pages. (There may be some overlap with "KVM guest
private memory" [2] here...)
Would definitely be keen on this MC!
JG
[1] https://lore.kernel.org/linux-mm/[email protected]/T/#m20c6df3d755cb79b6c426f7d9a02cec37d21e73e
[2] https://lore.kernel.org/lkml/[email protected]/T/
See James Morris's proposal here:
https://lore.kernel.org/all/[email protected]/
On 26/05/2023 04:36, James Morris wrote:
> [Side topic]
>
> Would folks be interested in a Linux Plumbers Conference MC on this
> topic generally, across different hypervisors, VMMs, and architectures?
>
> If so, please let me know who the key folk would be and we can try
writing
> up an MC proposal.
The fine-grain memory management proposal from James Gowans looks
interesting, especially the "side-car" virtual machines:
https://lore.kernel.org/all/[email protected]/
On 09/05/2023 11:55, Paolo Bonzini wrote:
> Hi all!
>
> We are planning on submitting a CFP to host a KVM Microconference at
> Linux Plumbers Conference 2023. To help justify the proposal, we would
> like to gather a list of folks that would likely attend, and crowdsource
> a list of topics to include in the proposal.
>
> For both this year and future years, the intent is that a KVM
> Microconference will complement KVM Forum, *NOT* supplant it. As you
> probably noticed, KVM Forum is going through a somewhat radical change in
> how it's organized; the conference is now free and (with some help from
> Red Hat) organized directly by the KVM and QEMU communities. Despite the
> unexpected changes and some teething pains, community response to KVM
> Forum continues to be overwhelmingly positive! KVM Forum will remain
> the venue of choice for KVM/userspace collaboration, for educational
> content covering both KVM and userspace, and to discuss new features in
> QEMU and other userspace projects.
>
> At least on the x86 side, however, the success of KVM Forum led us
> virtualization folks to operate in relative isolation. KVM depends on
> and impacts multiple subsystems (MM, scheduler, perf) in profound ways,
> and recently we’ve seen more and more ideas/features that require
> non-trivial changes outside KVM and buy-in from stakeholders that
> (typically) do not attend KVM Forum. Linux Plumbers Conference is a
> natural place to establish such collaboration within the kernel.
>
> Therefore, the aim of the KVM Microconference will be:
> * to provide a setting in which to discuss KVM and kernel internals
> * to increase collaboration and reduce friction with other subsystems
> * to discuss system virtualization issues that require coordination with
> other subsystems (such as VFIO, or guest support in arch/)
>
> Below is a rough draft of the planned CFP submission.
>
> Thanks!
>
> Paolo Bonzini (KVM Maintainer)
> Sean Christopherson (KVM x86 Co-Maintainer)
> Marc Zyngier (KVM ARM Co-Maintainer)
>
>
> ===================
> KVM Microconference
> ===================
>
> KVM (Kernel-based Virtual Machine) enables the use of hardware features
> to improve the efficiency, performance, and security of virtual machines
> created and managed by userspace. KVM was originally developed to host
> and accelerate "full" virtual machines running a traditional kernel and
> operating system, but has long since expanded to cover a wide array of use
> cases, e.g. hosting real time workloads, sandboxing untrusted workloads,
> deprivileging third party code, reducing the trusted computed base of
> security sensitive workloads, etc. As KVM's use cases have grown, so too
> have the requirements placed on KVM and the interactions between it and
> other kernel subsystems.
>
> The KVM Microconference will focus on how to evolve KVM and adjacent
> subsystems in order to satisfy new and upcoming requirements: serving
> guest memory that cannot be accessed by host userspace[1], providing
> accurate, feature-rich PMU/perf virtualization in cloud VMs[2], etc.
>
>
> Potential Topics:
> - Serving inaccessible/unmappable memory for KVM guests (protected VMs)
> - Optimizing mmu_notifiers, e.g. reducing TLB flushes and spurious zapping
> - Supporting multiple KVM modules (for non-disruptive upgrades)
> - Improving and hardening KVM+perf interactions
> - Implementing arch-agnostic abstractions in KVM (e.g. MMU)
> - Defining KVM requirements for hardware vendors
> - Utilizing "fault" injection to increase test coverage of edge cases
> - KVM vs VFIO (e.g. memory types, a rather hot topic on the ARM side)
>
>
> Key Attendees:
> - Paolo Bonzini <[email protected]> (KVM Maintainer)
> - Sean Christopherson <[email protected]> (KVM x86 Co-Maintainer)
> - Your name could be here!
>
> [1] https://lore.kernel.org/all/[email protected]
> [2] https://lore.kernel.org/all/CALMp9eRBOmwz=mspp0m5Q093K3rMUeAsF3vEL39MGV5Br9wEQQ@mail.gmail.com
>
>
On Fri, May 26, 2023, James Gowans wrote:
> On Tue, 2023-05-09 at 11:55 +0200, Paolo Bonzini wrote:
> > Hi all!
> >
> > We are planning on submitting a CFP to host a KVM Microconference at
> > Linux Plumbers Conference 2023. To help justify the proposal, we would
> > like to gather a list of folks that would likely attend, and crowdsource
> > a list of topics to include in the proposal.
>
> Hi Paolo,
>
> This MC sounds great! There are two topics I'd be keen to discuss, both in
> the KVM + memory-management realm:
>
> 1. Guest and kernel memory persistence across kexec for live update.
> Specifically focussing on the host IOMMU pgtable persistence for DMA-
> passthrough devices to support kexec while guest-driven DMA is still
> running. There is some discussion happening now about this [1] and
> hopefully the discussion and prototyping will continue in the run up to
> LPC.
I don't think a KVM MC conference would be the right venue for this discussion.
IIUC, KVM does not need to be involved in preserving guest memory or the IOMMU
page tables.
> 2. Supporting more fine-grain memory management and access control APIs
> for the virtualisation case specifically, for use-cases around live
> migration, memory oversubscription, and "side-car" virtual machines. These
> use cases would benefit from kernel support for things like dynamically
> updating IOMMU and MMU permissions independently at fine granularity, all
> without actually modifying the VMAs, to support fine-grain handling. And
> linking this topic to the one above: being able to do these things when
> not backed by struct pages. (There may be some overlap with "KVM guest
> private memory" [2] here...)
Yes, there's overlap with guest private memory. Though I actually think we should
start viewing it as "guest first" memory (I'm mentally thinking of it as guest_memfd()),
since there are potential benefits and applications beyond CoCo VMs for guest memory
that doesn't *need* to be mapped into host userspace. If the guest_memfd() idea comes
to fruition, then KVM would *need* a way to specify guest memory protections without
VMAs. So yes, definitely overlap :-)
If y'all are interested, guest_memfd() is the topic of discussion for the inaugural
KVM upstream call (PUCK)[*]. I would also be more than happy to carve out a PUCK
instance to discuss non-VMA-based MMU protections, i.e. we don't have to wait until
LPC to start hashing out the KVM API(s) and implementation.
[*] https://lore.kernel.org/all/[email protected]
On Fri, May 26, 2023, Sean Christopherson wrote:
> On Fri, May 26, 2023, James Gowans wrote:
> > On Tue, 2023-05-09 at 11:55 +0200, Paolo Bonzini wrote:
> > > Hi all!
> > >
> > > We are planning on submitting a CFP to host a KVM Microconference at
> > > Linux Plumbers Conference 2023. To help justify the proposal, we would
> > > like to gather a list of folks that would likely attend, and crowdsource
> > > a list of topics to include in the proposal.
> >
> > Hi Paolo,
> >
> > This MC sounds great! There are two topics I'd be keen to discuss, both in
> > the KVM + memory-management realm:
> >
> > 1. Guest and kernel memory persistence across kexec for live update.
> > Specifically focussing on the host IOMMU pgtable persistence for DMA-
> > passthrough devices to support kexec while guest-driven DMA is still
> > running. There is some discussion happening now about this [1] and
> > hopefully the discussion and prototyping will continue in the run up to
> > LPC.
>
> I don't think a KVM MC conference would be the right venue for this discussion.
> IIUC, KVM does not need to be involved in preserving guest memory or the IOMMU
> page tables.
Ah, I assume the KVM involvement comes from a potentially new filesystem for guest
memory?
5. More "advanced" memory management APIs/ioctls for virtualisation: Being
able to support things like DMA-driven post-copy live migration, memory
oversubscription, carving out chunks of memory from a VM to launch side-
car VMs, more fine-grain control of IOMMU or MMU permissions, etc. This
may be easier to achieve with a new filesystem, rather than coupling to
tempfs semantics and ioctls.
Hi,
What is the status of this microconference proposal? We'd be happy to
talk about Heki [1] and potentially other hypervisor supports.
Regards,
Mickaël
[1] https://lore.kernel.org/all/[email protected]/
On 26/05/2023 18:09, Mickaël Salaün wrote:
> See James Morris's proposal here:
> https://lore.kernel.org/all/[email protected]/
>
> On 26/05/2023 04:36, James Morris wrote:
> > [Side topic]
> >
> > Would folks be interested in a Linux Plumbers Conference MC on this
> > topic generally, across different hypervisors, VMMs, and architectures?
> >
> > If so, please let me know who the key folk would be and we can try
> writing
> > up an MC proposal.
>
> The fine-grain memory management proposal from James Gowans looks
> interesting, especially the "side-car" virtual machines:
> https://lore.kernel.org/all/[email protected]/
>
>
> On 09/05/2023 11:55, Paolo Bonzini wrote:
>> Hi all!
>>
>> We are planning on submitting a CFP to host a KVM Microconference at
>> Linux Plumbers Conference 2023. To help justify the proposal, we would
>> like to gather a list of folks that would likely attend, and crowdsource
>> a list of topics to include in the proposal.
>>
>> For both this year and future years, the intent is that a KVM
>> Microconference will complement KVM Forum, *NOT* supplant it. As you
>> probably noticed, KVM Forum is going through a somewhat radical change in
>> how it's organized; the conference is now free and (with some help from
>> Red Hat) organized directly by the KVM and QEMU communities. Despite the
>> unexpected changes and some teething pains, community response to KVM
>> Forum continues to be overwhelmingly positive! KVM Forum will remain
>> the venue of choice for KVM/userspace collaboration, for educational
>> content covering both KVM and userspace, and to discuss new features in
>> QEMU and other userspace projects.
>>
>> At least on the x86 side, however, the success of KVM Forum led us
>> virtualization folks to operate in relative isolation. KVM depends on
>> and impacts multiple subsystems (MM, scheduler, perf) in profound ways,
>> and recently we’ve seen more and more ideas/features that require
>> non-trivial changes outside KVM and buy-in from stakeholders that
>> (typically) do not attend KVM Forum. Linux Plumbers Conference is a
>> natural place to establish such collaboration within the kernel.
>>
>> Therefore, the aim of the KVM Microconference will be:
>> * to provide a setting in which to discuss KVM and kernel internals
>> * to increase collaboration and reduce friction with other subsystems
>> * to discuss system virtualization issues that require coordination with
>> other subsystems (such as VFIO, or guest support in arch/)
>>
>> Below is a rough draft of the planned CFP submission.
>>
>> Thanks!
>>
>> Paolo Bonzini (KVM Maintainer)
>> Sean Christopherson (KVM x86 Co-Maintainer)
>> Marc Zyngier (KVM ARM Co-Maintainer)
>>
>>
>> ===================
>> KVM Microconference
>> ===================
>>
>> KVM (Kernel-based Virtual Machine) enables the use of hardware features
>> to improve the efficiency, performance, and security of virtual machines
>> created and managed by userspace. KVM was originally developed to host
>> and accelerate "full" virtual machines running a traditional kernel and
>> operating system, but has long since expanded to cover a wide array of use
>> cases, e.g. hosting real time workloads, sandboxing untrusted workloads,
>> deprivileging third party code, reducing the trusted computed base of
>> security sensitive workloads, etc. As KVM's use cases have grown, so too
>> have the requirements placed on KVM and the interactions between it and
>> other kernel subsystems.
>>
>> The KVM Microconference will focus on how to evolve KVM and adjacent
>> subsystems in order to satisfy new and upcoming requirements: serving
>> guest memory that cannot be accessed by host userspace[1], providing
>> accurate, feature-rich PMU/perf virtualization in cloud VMs[2], etc.
>>
>>
>> Potential Topics:
>> - Serving inaccessible/unmappable memory for KVM guests (protected VMs)
>> - Optimizing mmu_notifiers, e.g. reducing TLB flushes and spurious zapping
>> - Supporting multiple KVM modules (for non-disruptive upgrades)
>> - Improving and hardening KVM+perf interactions
>> - Implementing arch-agnostic abstractions in KVM (e.g. MMU)
>> - Defining KVM requirements for hardware vendors
>> - Utilizing "fault" injection to increase test coverage of edge cases
>> - KVM vs VFIO (e.g. memory types, a rather hot topic on the ARM side)
>>
>>
>> Key Attendees:
>> - Paolo Bonzini <[email protected]> (KVM Maintainer)
>> - Sean Christopherson <[email protected]> (KVM x86 Co-Maintainer)
>> - Your name could be here!
>>
>> [1] https://lore.kernel.org/all/[email protected]
>> [2] https://lore.kernel.org/all/CALMp9eRBOmwz=mspp0m5Q093K3rMUeAsF3vEL39MGV5Br9wEQQ@mail.gmail.com
>>
>>
On Thu, Jun 01, 2023, Micka�l Sala�n wrote:
> Hi,
>
> What is the status of this microconference proposal? We'd be happy to talk
> about Heki [1] and potentially other hypervisor supports.
Proposal submitted (deadline is/was today), now we wait :-) IIUC, we should find
out rather quickly whether or not the KVM MC is a go.
On 6/7/23 12:13, Babis Chalios wrote:
>
>
> On our side, we 've been working on providing ways to let VMs (kernel
> and user space) know that they have been cloned/snapshotted/restored
> from snapshots[1].
>
> This is tightly coupled with PRNGs both in kernel and user space and
> there needs to be some collaboration with random.c to tie everything
> together [2][3]. It sounds like it could be a good fit for this MC
> (?).
Does the VM generation ID ACPI device from Microsoft work for you? It
is not tied to Hyper-V.
Paolo
On 9/5/23 11:55, Paolo Bonzini wrote:
> Hi all!
>
> We are planning on submitting a CFP to host a KVM Microconference at
> Linux Plumbers Conference 2023. To help justify the proposal, we would
> like to gather a list of folks that would likely attend, and crowdsource
> a list of topics to include in the proposal.
>
> For both this year and future years, the intent is that a KVM
> Microconference will complement KVM Forum, *NOT* supplant it. As you
> probably noticed, KVM Forum is going through a somewhat radical change in
> how it's organized; the conference is now free and (with some help from
> Red Hat) organized directly by the KVM and QEMU communities. Despite the
> unexpected changes and some teething pains, community response to KVM
> Forum continues to be overwhelmingly positive! KVM Forum will remain
> the venue of choice for KVM/userspace collaboration, for educational
> content covering both KVM and userspace, and to discuss new features in
> QEMU and other userspace projects.
>
> At least on the x86 side, however, the success of KVM Forum led us
> virtualization folks to operate in relative isolation. KVM depends on
> and impacts multiple subsystems (MM, scheduler, perf) in profound ways,
> and recently we’ve seen more and more ideas/features that require
> non-trivial changes outside KVM and buy-in from stakeholders that
> (typically) do not attend KVM Forum. Linux Plumbers Conference is a
> natural place to establish such collaboration within the kernel.
>
> Therefore, the aim of the KVM Microconference will be:
> * to provide a setting in which to discuss KVM and kernel internals
> * to increase collaboration and reduce friction with other subsystems
> * to discuss system virtualization issues that require coordination with
> other subsystems (such as VFIO, or guest support in arch/)
>
> Below is a rough draft of the planned CFP submission.
>
> Thanks!
>
> Paolo Bonzini (KVM Maintainer)
> Sean Christopherson (KVM x86 Co-Maintainer)
> Marc Zyngier (KVM ARM Co-Maintainer)
>
>
> ===================
> KVM Microconference
> ===================
>
> KVM (Kernel-based Virtual Machine) enables the use of hardware features
> to improve the efficiency, performance, and security of virtual machines
> created and managed by userspace. KVM was originally developed to host
> and accelerate "full" virtual machines running a traditional kernel and
> operating system, but has long since expanded to cover a wide array of
> use
> cases, e.g. hosting real time workloads, sandboxing untrusted workloads,
> deprivileging third party code, reducing the trusted computed base of
> security sensitive workloads, etc. As KVM's use cases have grown, so too
> have the requirements placed on KVM and the interactions between it and
> other kernel subsystems.
>
> The KVM Microconference will focus on how to evolve KVM and adjacent
> subsystems in order to satisfy new and upcoming requirements: serving
> guest memory that cannot be accessed by host userspace[1], providing
> accurate, feature-rich PMU/perf virtualization in cloud VMs[2], etc.
>
>
> Potential Topics:
> - Serving inaccessible/unmappable memory for KVM guests (protected VMs)
> - Optimizing mmu_notifiers, e.g. reducing TLB flushes and spurious
> zapping
> - Supporting multiple KVM modules (for non-disruptive upgrades)
> - Improving and hardening KVM+perf interactions
> - Implementing arch-agnostic abstractions in KVM (e.g. MMU)
> - Defining KVM requirements for hardware vendors
> - Utilizing "fault" injection to increase test coverage of edge cases
> - KVM vs VFIO (e.g. memory types, a rather hot topic on the ARM side)
>
>
> Key Attendees:
> - Paolo Bonzini <[email protected]> (KVM Maintainer)
> - Sean Christopherson <[email protected]> (KVM x86 Co-Maintainer)
> - Your name could be here!
>
> [1]
> https://lore.kernel.org/all/[email protected]
> [2]
> https://lore.kernel.org/all/CALMp9eRBOmwz=mspp0m5Q093K3rMUeAsF3vEL39MGV5Br9wEQQ@mail.gmail.com
>
Hi Paolo,
I think this idea is great!
On our side, we 've been working on providing ways to let VMs (kernel
and user space)
know that they have been cloned/snapshotted/restored from snapshots[1].
This is tightly coupled with PRNGs both in kernel and user space and
there needs to be some
collaboration with random.c to tie everything together [2][3]. It sounds
like it could be a good
fit for this MC (?).
It would be interested to figure out whether such notifications would be
interested to other
parts of the kernel as well.
Cheers,
Babis
[1] https://www.spinics.net/lists/kernel/msg4808187.html
[2]
https://www.mail-archive.com/[email protected]/msg09016.html
[3]
https://lore.kernel.org/lkml/[email protected]/T/
On 7/6/23 12:37, Paolo Bonzini wrote:
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender
> and know the content is safe.
>
>
>
> On 6/7/23 12:13, Babis Chalios wrote:
>>
>>
>> On our side, we 've been working on providing ways to let VMs (kernel
>> and user space) know that they have been cloned/snapshotted/restored
>> from snapshots[1].
>>
>> This is tightly coupled with PRNGs both in kernel and user space and
>> there needs to be some collaboration with random.c to tie everything
>> together [2][3]. It sounds like it could be a good fit for this MC
>> (?).
>
> Does the VM generation ID ACPI device from Microsoft work for you? It
> is not tied to Hyper-V.
>
> Paolo
>
It's a long story :P
TLDR:
For some use-cases it should be enough [1]. For PRNGs though it doesn't
seem to work as is. (1) The Linux implementation of VMGENID is racy and
(2) last year there were long discussions related to the user-space
ABI, which diverged us away from it.
BTW, just saw that you have already submitted the proposal for the MC,
sorry for being late :)
Cheers,
Babis