by Christoffer Dall

[permalink] [raw]

Subject: Re: [PATCH RFC 0/7] Support KVM being compiled as a kernel module on arm64

On Fri, Oct 25, 2019 at 10:48:46AM +0800, Shannon Zhao wrote:
>
>
> On 2019/10/24 21:41, Marc Zyngier wrote:
> > On 2019-10-24 11:58, James Morse wrote:
> > > Hi Shannon,
> > >
> > > On 24/10/2019 11:27, Shannon Zhao wrote:
> > > > Curently KVM ARM64 doesn't support to compile as a kernel module. It's
> > > > useful to compile KVM as a module.
> > >
> > > > For example, it could reload kvm without rebooting host machine.
> > >
> > > What problem does this solve?
> > >
> > > KVM has some funny requirements that aren't normal for a module. On
> > > v8.0 hardware it must
> > > have an idmap. Modules don't usually expect their code to be
> > > physically contiguous, but
> > > KVM does. KVM influences they way some of the irqchip stuff is set up
> > > during early boot
> > > (EOI mode ... not that I understand it).
> >
> > We change the EOImode solely based on how we were booted (EL2 or not).
> > KVM doesn't directly influences that (it comes in the picture much
> > later).
> >
> > > (I think KVM-as-a-module on x86 is an artifact of how it was developed)
> > >
> > >
> > > > This patchset support this feature while there are some limitations
> > > > to be solved. But I just send it out as RFC to get more suggestion and
> > > > comments.
> > >
> > > > Curently it only supports for VHE system due to the hyp code section
> > > > address variables like __hyp_text_start.
> > >
> > > We still need to support !VHE systems, and we need to do it with a
> > > single image.
> > >
> > >
> > > > Also it can't call
> > > > kvm_update_va_mask when loading kvm module and kernel panic with below
> > > > errors. So I make kern_hyp_va into a nop funtion.
> > >
> > > Making this work for the single-Image on v8.0 is going to be a
> > > tremendous amount of work.
> > > What is the payoff?
> >
> > I can only agree. !VHE is something we're going to support for the
> > foreseeable
> > future (which is roughly equivalent to "forever"), and modules have
> > properties
> > that are fundamentally incompatible with the way KVM works with !VHE.
> >
> Yes, with this patchset we still support !VHE system with built-in KVM.
> While for VHE system we could support kernel module and check at module init
> to avoid wrong usage of kvm module on !VHE systems.
>
> > If the only purpose of this work is to be able to swap KVM implementations
> > in a development environment, then it really isn't worth the effort.
> >
> Making KVM as a kernel module has many advantages both for development and
> real use environment. For example, we can backport and update KVM codes
> independently and don't need to recompile kernel. Also making KVM as a
> kernel module is a basic for kvm hot upgrade feature without shutdown VMs
> and hosts. This is very important for Cloud Service Provider to provides
> non-stop services for its customers.
>
But KVM on arm64 is pretty intertwined with the rest of the kernel, and
things like the arch timers, for example, really depend on the exact
semantics of how the rest of the kernel changes. I fear that you'd end
up back-porting patches that depend on changes to irqchip and timers in
the core code, and you'll get even more oddly-defined behavior in the
wild.

How would you manage that, and how would the end result be a more stable
environment than what you have today?

Also, I'm curious if you expect to find more bugs in the hypervisor
itself than in the rest of the kernel, because it's only in the former
case you can avoid a reboot of the host, and all things considered this
would appear to only help in a small fraction of the cases where you
have to patch things?

Thanks,

Christoffer

2019-10-25 19:43:33

by Marc Zyngier

[permalink] [raw]

Subject: Re: [PATCH RFC 0/7] Support KVM being compiled as a kernel module on arm64

On 2019-10-25 03:48, Shannon Zhao wrote:
> On 2019/10/24 21:41, Marc Zyngier wrote:
>> On 2019-10-24 11:58, James Morse wrote:
>>> Hi Shannon,
>>>
>>> On 24/10/2019 11:27, Shannon Zhao wrote:
>>>> Curently KVM ARM64 doesn't support to compile as a kernel module.
>>>> It's
>>>> useful to compile KVM as a module.
>>>
>>>> For example, it could reload kvm without rebooting host machine.
>>>
>>> What problem does this solve?
>>>
>>> KVM has some funny requirements that aren't normal for a module. On
>>> v8.0 hardware it must
>>> have an idmap. Modules don't usually expect their code to be
>>> physically contiguous, but
>>> KVM does. KVM influences they way some of the irqchip stuff is set
>>> up
>>> during early boot
>>> (EOI mode ... not that I understand it).
>> We change the EOImode solely based on how we were booted (EL2 or
>> not).
>> KVM doesn't directly influences that (it comes in the picture much
>> later).
>>
>>> (I think KVM-as-a-module on x86 is an artifact of how it was
>>> developed)
>>>
>>>
>>>> This patchset support this feature while there are some
>>>> limitations
>>>> to be solved. But I just send it out as RFC to get more suggestion
>>>> and
>>>> comments.
>>>
>>>> Curently it only supports for VHE system due to the hyp code
>>>> section
>>>> address variables like __hyp_text_start.
>>>
>>> We still need to support !VHE systems, and we need to do it with a
>>> single image.
>>>
>>>
>>>> Also it can't call
>>>> kvm_update_va_mask when loading kvm module and kernel panic with
>>>> below
>>>> errors. So I make kern_hyp_va into a nop funtion.
>>>
>>> Making this work for the single-Image on v8.0 is going to be a
>>> tremendous amount of work.
>>> What is the payoff?
>> I can only agree. !VHE is something we're going to support for the
>> foreseeable
>> future (which is roughly equivalent to "forever"), and modules have
>> properties
>> that are fundamentally incompatible with the way KVM works with
>> !VHE.
>
> Yes, with this patchset we still support !VHE system with built-in
> KVM. While for VHE system we could support kernel module and check at
> module init to avoid wrong usage of kvm module on !VHE systems.

How do you reconcile this with the need to have a single kernel image
that
is full-featured on all architecture revisions? In your view of the
world,
it is OK to have a VHE-specific kernel that won't have KVM on a v8.0
system.
From my point of view, this simply isn't acceptable.

>> If the only purpose of this work is to be able to swap KVM
>> implementations
>> in a development environment, then it really isn't worth the effort.
>
> Making KVM as a kernel module has many advantages both for
> development and real use environment. For example, we can backport and
> update KVM codes independently and don't need to recompile kernel. Also
> making KVM as a kernel module is a basic for kvm hot upgrade feature
> without shutdown VMs and hosts. This is very important for Cloud
> Service Provider to provides non-stop services for its customers.

The goal is certainly commendable.

But pretending that KVM is fully independent from the rest of the
kernel is
an intellectual fallacy. Most of the issues that affect KVM at any
given time
are actually core kernel issues that cannot be solved by just updating
the
KVM module. You need to update the whole thing, not just the tiny bit
that
switches between host and guests.

M.
--
Jazz is not dead. It just smells funny...