[Reposting it here from Linaro Open Discussion List for more eyes to look at]
Hello,
I have recently started to dabble with ARM CCA stuff and check if our
recent changes to support vCPU Hotplug in ARM64 can work in the realm
world. I have realized that in the RMM specification[1] PSCI_CPU_ON
command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
from the host. This might be required to support vCPU Hotplug feature
in the realm world in future. vCPU Hotplug is an important feature to
support kata-containers in realm world as it reduces the VM boot time
and facilitates dynamic adjustment of vCPUs (which I think should be
true even with Realm world as current implementation only makes use
of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
As per our recent changes [2], [3] related to support vCPU Hotplug on
ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
PSCI_CPU_ON should undergo similar policy checks and I think,
1. Host should *deny* to online the target vCPUs which are NOT plugged
2. This means target REC should be denied by host. Can host call
RMI_PSCI_COMPETE in such s case?
3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
4. Failure condition (B5.3.3.2) should be amended with
runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
post: result == PSCI_DENIED (?)
5. Change would also be required in the flow (D1.4 PSCI flows) depicting
PSCI_CPU_ON flow (D1.4.1)
I do understand that ARM CCA support is in its infancy stage and
discussing about vCPU Hotplug in realm world seem to be a far-fetched
idea right now. But specification changes require lot of time and if
this change is really required then it should be further discussed
within ARM.
Many thanks!
Bes regards
Salil
References:
[1] https://developer.arm.com/documentation/den0137/latest/
[2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
[3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
Hi Salil
Thanks for raising this.
On 19/07/2023 03:35, Salil Mehta wrote:
> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
>
> Hello,
> I have recently started to dabble with ARM CCA stuff and check if our
> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> from the host. This might be required to support vCPU Hotplug feature
> in the realm world in future. vCPU Hotplug is an important feature to
> support kata-containers in realm world as it reduces the VM boot time
> and facilitates dynamic adjustment of vCPUs (which I think should be
> true even with Realm world as current implementation only makes use
> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>
>
> As per our recent changes [2], [3] related to support vCPU Hotplug on
> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> PSCI_CPU_ON should undergo similar policy checks and I think,
>
> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> 2. This means target REC should be denied by host. Can host call
> RMI_PSCI_COMPETE in such s case?
> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
The Realm exit with EXIT_PSCI already provides the parameters passed
onto the PSCI request. This happens for all PSCI calls except
(PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses
the request (wherever applicable).
However, the RMM spec currently doesn't allow denying the request.
i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
We will address this in the RMM spec and get back to you.
Kind regards
Suzuki
> 4. Failure condition (B5.3.3.2) should be amended with
> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> post: result == PSCI_DENIED (?)
> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> PSCI_CPU_ON flow (D1.4.1)
>
>
> I do understand that ARM CCA support is in its infancy stage and
> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> idea right now. But specification changes require lot of time and if
> this change is really required then it should be further discussed
> within ARM.
>
> Many thanks!
>
>
> Bes regards
> Salil
>
>
> References:
>
> [1] https://developer.arm.com/documentation/den0137/latest/
> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
>
Hi Salil
On 19/07/2023 10:28, Suzuki K Poulose wrote:
> Hi Salil
>
> Thanks for raising this.
>
> On 19/07/2023 03:35, Salil Mehta wrote:
>> [Reposting it here from Linaro Open Discussion List for more eyes to
>> look at]
>>
>> Hello,
>> I have recently started to dabble with ARM CCA stuff and check if our
>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
>> from the host. This might be required to support vCPU Hotplug feature
>> in the realm world in future. vCPU Hotplug is an important feature to
>> support kata-containers in realm world as it reduces the VM boot time
>> and facilitates dynamic adjustment of vCPUs (which I think should be
>> true even with Realm world as current implementation only makes use
>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>>
>>
>> As per our recent changes [2], [3] related to support vCPU Hotplug on
>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
>> PSCI_CPU_ON should undergo similar policy checks and I think,
>>
>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
>> 2. This means target REC should be denied by host. Can host call
>> RMI_PSCI_COMPETE in such s case?
>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
>
> The Realm exit with EXIT_PSCI already provides the parameters passed
> onto the PSCI request. This happens for all PSCI calls except
> (PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
> the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses
> the request (wherever applicable).
>
> However, the RMM spec currently doesn't allow denying the request.
> i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
> We will address this in the RMM spec and get back to you.
This is now resolved in RMMv1.0-eac3 spec, available here [0].
This allows the host to DENY a PSCI_CPU_ON request. The RMM ensures that
the response doesn't violate the security guarantees by checking the
state of the target REC.
[0] https://developer.arm.com/documentation/den0137/latest/
Kind regards
Suzuki
>
> Kind regards
> Suzuki
>
>
>> 4. Failure condition (B5.3.3.2) should be amended with
>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
>> post: result == PSCI_DENIED (?)
>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
>> PSCI_CPU_ON flow (D1.4.1)
>>
>> I do understand that ARM CCA support is in its infancy stage and
>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
>> idea right now. But specification changes require lot of time and if
>> this change is really required then it should be further discussed
>> within ARM.
>>
>> Many thanks!
>>
>>
>> Bes regards
>> Salil
>>
>>
>> References:
>>
>> [1] https://developer.arm.com/documentation/den0137/latest/
>> [2] https://github.com/salil-mehta/qemu.git
>> virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git
>> virtual_cpu_hotplug/rfc/v2
>>
>
Is vcpu hotplug supported in arm virt-machine now?
There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
I do not know why it is not merged. I ask this question because we propose
similar patch about LoongArch system in qemu mailing list, and kernel need not be
modified for vcpu hotplug.
Regards
Bibo, mao
在 2023/7/19 10:35, Salil Mehta 写道:
> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
>
> Hello,
> I have recently started to dabble with ARM CCA stuff and check if our
> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> from the host. This might be required to support vCPU Hotplug feature
> in the realm world in future. vCPU Hotplug is an important feature to
> support kata-containers in realm world as it reduces the VM boot time
> and facilitates dynamic adjustment of vCPUs (which I think should be
> true even with Realm world as current implementation only makes use
> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>
>
> As per our recent changes [2], [3] related to support vCPU Hotplug on
> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> PSCI_CPU_ON should undergo similar policy checks and I think,
>
> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> 2. This means target REC should be denied by host. Can host call
> RMI_PSCI_COMPETE in such s case?
> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> 4. Failure condition (B5.3.3.2) should be amended with
> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> post: result == PSCI_DENIED (?)
> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> PSCI_CPU_ON flow (D1.4.1)
>
>
> I do understand that ARM CCA support is in its infancy stage and
> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> idea right now. But specification changes require lot of time and if
> this change is really required then it should be further discussed
> within ARM.
>
> Many thanks!
>
>
> Bes regards
> Salil
>
>
> References:
>
> [1] https://developer.arm.com/documentation/den0137/latest/
> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
Hi Suzuki,
Sorry for replying late as I was on/off last week to undergo some medical test.
> From: Suzuki K Poulose <[email protected]>
> Sent: Monday, July 24, 2023 5:27 PM
>
> Hi Salil
>
> On 19/07/2023 10:28, Suzuki K Poulose wrote:
> > Hi Salil
> >
> > Thanks for raising this.
> >
> > On 19/07/2023 03:35, Salil Mehta wrote:
> >> [Reposting it here from Linaro Open Discussion List for more eyes to
> >> look at]
> >>
> >> Hello,
> >> I have recently started to dabble with ARM CCA stuff and check if our
> >> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> >> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> >> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> >> from the host. This might be required to support vCPU Hotplug feature
> >> in the realm world in future. vCPU Hotplug is an important feature to
> >> support kata-containers in realm world as it reduces the VM boot time
> >> and facilitates dynamic adjustment of vCPUs (which I think should be
> >> true even with Realm world as current implementation only makes use
> >> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >>
> >>
> >> As per our recent changes [2], [3] related to support vCPU Hotplug on
> >> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> >> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> >> PSCI_CPU_ON should undergo similar policy checks and I think,
> >>
> >> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> >> 2. This means target REC should be denied by host. Can host call
> >> RMI_PSCI_COMPETE in such s case?
> >> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> >
> > The Realm exit with EXIT_PSCI already provides the parameters passed
> > onto the PSCI request. This happens for all PSCI calls except
> > (PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
> > the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses
> > the request (wherever applicable).
> >
> > However, the RMM spec currently doesn't allow denying the request.
> > i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
> > We will address this in the RMM spec and get back to you.
>
> This is now resolved in RMMv1.0-eac3 spec, available here [0].
>
> This allows the host to DENY a PSCI_CPU_ON request. The RMM ensures that
> the response doesn't violate the security guarantees by checking the
> state of the target REC.
>
> [0] https://developer.arm.com/documentation/den0137/latest/
Many thanks for taking this up proactively and getting it done as well
very efficiently. Really appreciate this!
I acknowledge below new changes part of the newly released RMM
Specification [3] (Page-2) (Release Information 1.0-eac3 20-07-2023):
1. Addition of B2.19 PsciReturnCodePermitted function [3] (Page-126)
2. Addition of 'status' in B3.3.7.2 Failure conditions of the
B3.3.7 RMI_PSCI_COMPLETE command [3] (Page-160)
Some Further Suggestions:
1. It would be really helpful if PSCI_DENIED can be accommodated somewhere
in the flow diagram (D1.4.1 PSCI_CPU_ON flow) [3] (Page-297) as well.
2. You would need changes to handle the return value of the PSCI_DENIED
in this below patch [2] as well from ARM CCA series [1]
@James, Any further thoughts on this?
References:
[1] [RFC PATCH 00/28] arm64: Support for Arm CCA in KVM
[2] [RFC PATCH 19/28] KVM: arm64: Validate register access for a Realm VM
https://lore.kernel.org/lkml/[email protected]/T/#m6c10b9a27c4a724967c1800facacaa9443b38b4c
[3] ARM Realm Management Monitor specification(DEN0137 1.0-eac3)
https://developer.arm.com/documentation/den0137/latest/
Thanks
Salil.
> >> 4. Failure condition (B5.3.3.2) should be amended with
> >> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >> post: result == PSCI_DENIED (?)
> >> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >> PSCI_CPU_ON flow (D1.4.1)
> >>
> >> I do understand that ARM CCA support is in its infancy stage and
> >> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> >> idea right now. But specification changes require lot of time and if
> >> this change is really required then it should be further discussed
> >> within ARM.
> >>
> >> Many thanks!
> >>
> >>
> >> Bes regards
> >> Salil
> >>
> >>
> >> References:
> >>
> >> [1] https://developer.arm.com/documentation/den0137/latest/
> >> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
Hi Bibo,
> From: bibo mao <[email protected]>
> Sent: Tuesday, July 25, 2023 1:29 AM
> To: Salil Mehta <[email protected]>
> Cc: Catalin Marinas <[email protected]>; Jonathan Cameron
> <[email protected]>; Marc Zyngier <[email protected]>; Will Deacon
> <[email protected]>; [email protected]; [email protected];
> [email protected]; [email protected]; Salil Mehta
> <[email protected]>; [email protected]; yuzenghui
> <[email protected]>; [email protected]; linux-arm-
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; Suzuki K Poulose <[email protected]>; Jean-
> Philippe Brucker <[email protected]>; [email protected]; linux-
> [email protected]
> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world
> might require ARM spec change?
>
> Is vcpu hotplug supported in arm virt-machine now?
Not yet. We are working on it. Please check the RFCs being tested.
[1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
[2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
> I do not know why it is not merged.
I think you are referring to patches [3], [4]? Please follow the discussion
for details.
[3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
https://lore.kernel.org/all/[email protected]/
[4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
https://lore.kernel.org/all/[email protected]/#r
In summary, there were some ARM64 Architecture constraints which were being
violated in the earlier patches of the kernel [4] so we had to re-think of the
kernel changes. The Qemu part mostly remains same with some new introductions
of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
in VMM/Qemu.
You can follow the KVMForum conference presentations [5], [6] delivered in the
year 2020 and 2023 to get hold of more details related to this.
[5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
[6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
https://kvmforum2020.sched.com/event/eE4m
> I ask this question because we propose
> similar patch about LoongArch system in qemu mailing list, and kernel need not be
> modified for vcpu hotplug.
Could you please share the link of your patches so that we can have a look and
draw a comparison?
Thanks
Salil.
>
> Regards
> Bibo, mao
>
> 在 2023/7/19 10:35, Salil Mehta 写道:
> > [Reposting it here from Linaro Open Discussion List for more eyes to look
> at]
> >
> > Hello,
> > I have recently started to dabble with ARM CCA stuff and check if our
> > recent changes to support vCPU Hotplug in ARM64 can work in the realm
> > world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> > command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> > from the host. This might be required to support vCPU Hotplug feature
> > in the realm world in future. vCPU Hotplug is an important feature to
> > support kata-containers in realm world as it reduces the VM boot time
> > and facilitates dynamic adjustment of vCPUs (which I think should be
> > true even with Realm world as current implementation only makes use
> > of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >
> >
> > As per our recent changes [2], [3] related to support vCPU Hotplug on
> > ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> > user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> > PSCI_CPU_ON should undergo similar policy checks and I think,
> >
> > 1. Host should *deny* to online the target vCPUs which are NOT plugged
> > 2. This means target REC should be denied by host. Can host call
> > RMI_PSCI_COMPETE in such s case?
> > 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> > 4. Failure condition (B5.3.3.2) should be amended with
> > runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> > post: result == PSCI_DENIED (?)
> > 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> > PSCI_CPU_ON flow (D1.4.1)
> >
> >
> > I do understand that ARM CCA support is in its infancy stage and
> > discussing about vCPU Hotplug in realm world seem to be a far-fetched
> > idea right now. But specification changes require lot of time and if
> > this change is really required then it should be further discussed
> > within ARM.
> >
> > Many thanks!
> >
> >
> > Bes regards
> > Salil
> >
> >
> > References:
> >
> > [1] https://developer.arm.com/documentation/den0137/latest/
> > [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> > [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
在 2023/7/25 08:56, Salil Mehta 写道:
> Hi Bibo,
>
>> From: bibo mao <[email protected]>
>> Sent: Tuesday, July 25, 2023 1:29 AM
>> To: Salil Mehta <[email protected]>
>> Cc: Catalin Marinas <[email protected]>; Jonathan Cameron
>> <[email protected]>; Marc Zyngier <[email protected]>; Will Deacon
>> <[email protected]>; [email protected]; [email protected];
>> [email protected]; [email protected]; Salil Mehta
>> <[email protected]>; [email protected]; yuzenghui
>> <[email protected]>; [email protected]; linux-arm-
>> [email protected]; [email protected];
>> [email protected]; [email protected]; [email protected];
>> [email protected]; Suzuki K Poulose <[email protected]>; Jean-
>> Philippe Brucker <[email protected]>; [email protected]; linux-
>> [email protected]
>> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world
>> might require ARM spec change?
>>
>> Is vcpu hotplug supported in arm virt-machine now?
>
> Not yet. We are working on it. Please check the RFCs being tested.
>
>
> [1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
> https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> [2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
> https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
>
>
>> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
>> I do not know why it is not merged.
>
>
> I think you are referring to patches [3], [4]? Please follow the discussion
> for details.
yeap, we reference the patch [3], [4] and benefit from them greatly -:)
The patch for LoongArch vcpu hotplug link is:
https://lore.kernel.org/qemu-devel/[email protected]/T/#t
Regards
Bibo Mao
>
>
> [3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
> https://lore.kernel.org/all/[email protected]/
> [4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
> https://lore.kernel.org/all/[email protected]/#r
>
>
> In summary, there were some ARM64 Architecture constraints which were being
> violated in the earlier patches of the kernel [4] so we had to re-think of the
> kernel changes. The Qemu part mostly remains same with some new introductions
> of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
> in VMM/Qemu.
>
>
> You can follow the KVMForum conference presentations [5], [6] delivered in the
> year 2020 and 2023 to get hold of more details related to this.
>
>
> [5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
> https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
> [6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
> https://kvmforum2020.sched.com/event/eE4m
>
>
>
>> I ask this question because we propose
>> similar patch about LoongArch system in qemu mailing list, and kernel need not be
>> modified for vcpu hotplug.
>
>
> Could you please share the link of your patches so that we can have a look and
> draw a comparison?
>
>
> Thanks
> Salil.
>
>>
>> Regards
>> Bibo, mao
>>
>> 在 2023/7/19 10:35, Salil Mehta 写道:
>>> [Reposting it here from Linaro Open Discussion List for more eyes to look
>> at]
>>>
>>> Hello,
>>> I have recently started to dabble with ARM CCA stuff and check if our
>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
>>> from the host. This might be required to support vCPU Hotplug feature
>>> in the realm world in future. vCPU Hotplug is an important feature to
>>> support kata-containers in realm world as it reduces the VM boot time
>>> and facilitates dynamic adjustment of vCPUs (which I think should be
>>> true even with Realm world as current implementation only makes use
>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>>>
>>>
>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
>>> PSCI_CPU_ON should undergo similar policy checks and I think,
>>>
>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
>>> 2. This means target REC should be denied by host. Can host call
>>> RMI_PSCI_COMPETE in such s case?
>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
>>> 4. Failure condition (B5.3.3.2) should be amended with
>>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
>>> post: result == PSCI_DENIED (?)
>>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
>>> PSCI_CPU_ON flow (D1.4.1)
>>>
>>>
>>> I do understand that ARM CCA support is in its infancy stage and
>>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
>>> idea right now. But specification changes require lot of time and if
>>> this change is really required then it should be further discussed
>>> within ARM.
>>>
>>> Many thanks!
>>>
>>>
>>> Bes regards
>>> Salil
>>>
>>>
>>> References:
>>>
>>> [1] https://developer.arm.com/documentation/den0137/latest/
>>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
>>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
Hello,
> From: bibo mao <[email protected]>
> Sent: Tuesday, July 25, 2023 2:14 AM
> To: Salil Mehta <[email protected]>
[...]
> 在 2023/7/25 08:56, Salil Mehta 写道:
> > Hi Bibo,
> >
> >> From: bibo mao <[email protected]>
> >> Sent: Tuesday, July 25, 2023 1:29 AM
> >> To: Salil Mehta <[email protected]>
> >> Cc: Catalin Marinas <[email protected]>; Jonathan Cameron
> >> <[email protected]>; Marc Zyngier <[email protected]>; Will Deacon
> >> <[email protected]>; [email protected]; [email protected];
> >> [email protected]; [email protected]; Salil Mehta
> >> <[email protected]>; [email protected]; yuzenghui
> >> <[email protected]>; [email protected]; linux-arm-
> >> [email protected]; [email protected];
> >> [email protected]; [email protected]; [email protected];
> >> [email protected]; Suzuki K Poulose <[email protected]>; Jean-
> >> Philippe Brucker <[email protected]>; [email protected]; [email protected]
> >> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?
> >>
> >> Is vcpu hotplug supported in arm virt-machine now?
> >
> > Not yet. We are working on it. Please check the RFCs being tested.
> >
> >
> > [1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
> > https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> > [2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
> > https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
> >
> >
> >> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
> >> I do not know why it is not merged.
> >
> >
> > I think you are referring to patches [3], [4]? Please follow the discussion
> > for details.
>
> yeap, we reference the patch [3], [4] and benefit from them greatly -:)
I am glad that our current work is useful to more than one architecture and it
was one of the aim of our work as well but...
> The patch for LoongArch vcpu hotplug link is:
> https://lore.kernel.org/qemu-devel/[email protected]/T/#t
I quickly went through above patches and it looks like this patch-set is mostly
based on our latest patches which are at [1], [2] and not just at [3], [4]. As I
could see most of the functions which you have ported to your architecture are
part of our Qemu repositories [2] which we have yet to push to community. As I
am working towards RFC V2 patches and which shall be floated soon. It does not
makes sense for you to duplicate the GED/ACPI changes which are common across
architectures and which have been derived from the ARM64 vCPU Hotplug support
original patches.
This will create merge conflicts, will break large part of our original patch-set.
Hence, I would request you to drop patches 1-4 from your patch-set or rebase
it over ARM64 original patches in a week or 2 week of time. This is to avoid
spoiling our previous years of hard work for the topic we have been persistently
making efforts as you can see through the code and our detailed presentations.
Hope you will agree we all need to respect others efforts and time in this
mode of open-source collaboration.
Rest assured I will help you in review of your architecture specific patch-set
as it is a work of mutual interest.
Thanks for understanding!
Best Wishes,
Salil.
> Regards
> Bibo Mao
>
> >
> >
> > [3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
> > https://lore.kernel.org/all/[email protected]/
> > [4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
> > https://lore.kernel.org/all/[email protected]/#r
> >
> >
> > In summary, there were some ARM64 Architecture constraints which were being
> > violated in the earlier patches of the kernel [4] so we had to re-think of the
> > kernel changes. The Qemu part mostly remains same with some new introductions
> > of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
> > in VMM/Qemu.
> >
> >
> > You can follow the KVMForum conference presentations [5], [6] delivered in the
> > year 2020 and 2023 to get hold of more details related to this.
> >
> >
> > [5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
> > https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
> > [6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
> > https://kvmforum2020.sched.com/event/eE4m
> >
> >
> >
> >> I ask this question because we propose
> >> similar patch about LoongArch system in qemu mailing list, and kernel need not be
> >> modified for vcpu hotplug.
> >
> >
> > Could you please share the link of your patches so that we can have a look and
> > draw a comparison?
> >
> >
> > Thanks
> > Salil.
> >
> >>
> >> Regards
> >> Bibo, mao
> >>
> >> 在 2023/7/19 10:35, Salil Mehta 写道:
> >>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
> >>>
> >>> Hello,
> >>> I have recently started to dabble with ARM CCA stuff and check if our
> >>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> >>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> >>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> >>> from the host. This might be required to support vCPU Hotplug feature
> >>> in the realm world in future. vCPU Hotplug is an important feature to
> >>> support kata-containers in realm world as it reduces the VM boot time
> >>> and facilitates dynamic adjustment of vCPUs (which I think should be
> >>> true even with Realm world as current implementation only makes use
> >>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >>>
> >>>
> >>> As per our recent changes [2], [3] related to support vCPU Hotplug on
> >>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> >>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> >>> PSCI_CPU_ON should undergo similar policy checks and I think,
> >>>
> >>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> >>> 2. This means target REC should be denied by host. Can host call
> >>> RMI_PSCI_COMPETE in such s case?
> >>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> >>> 4. Failure condition (B5.3.3.2) should be amended with
> >>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >>> post: result == PSCI_DENIED (?)
> >>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >>> PSCI_CPU_ON flow (D1.4.1)
> >>>
> >>>
> >>> I do understand that ARM CCA support is in its infancy stage and
> >>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> >>> idea right now. But specification changes require lot of time and if
> >>> this change is really required then it should be further discussed
> >>> within ARM.
> >>>
> >>> Many thanks!
> >>>
> >>>
> >>> Bes regards
> >>> Salil
> >>>
> >>>
> >>> References:
> >>>
> >>> [1] https://developer.arm.com/documentation/den0137/latest/
> >>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
在 2023/7/25 13:45, Salil Mehta 写道:
> Hello,
>
>> From: bibo mao <[email protected]>
>> Sent: Tuesday, July 25, 2023 2:14 AM
>> To: Salil Mehta <[email protected]>
>
>
> [...]
>
>
>> 在 2023/7/25 08:56, Salil Mehta 写道:
>>> Hi Bibo,
>>>
>>>> From: bibo mao <[email protected]>
>>>> Sent: Tuesday, July 25, 2023 1:29 AM
>>>> To: Salil Mehta <[email protected]>
>>>> Cc: Catalin Marinas <[email protected]>; Jonathan Cameron
>>>> <[email protected]>; Marc Zyngier <[email protected]>; Will Deacon
>>>> <[email protected]>; [email protected]; [email protected];
>>>> [email protected]; [email protected]; Salil Mehta
>>>> <[email protected]>; [email protected]; yuzenghui
>>>> <[email protected]>; [email protected]; linux-arm-
>>>> [email protected]; [email protected];
>>>> [email protected]; [email protected]; [email protected];
>>>> [email protected]; Suzuki K Poulose <[email protected]>; Jean-
>>>> Philippe Brucker <[email protected]>; [email protected]; [email protected]
>>>> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?
>>>>
>>>> Is vcpu hotplug supported in arm virt-machine now?
>>>
>>> Not yet. We are working on it. Please check the RFCs being tested.
>>>
>>>
>>> [1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
>>> https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
>>> [2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
>>> https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
>>>
>>>
>>>> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
>>>> I do not know why it is not merged.
>>>
>>>
>>> I think you are referring to patches [3], [4]? Please follow the discussion
>>> for details.
>>
>> yeap, we reference the patch [3], [4] and benefit from them greatly -:)
>
>
> I am glad that our current work is useful to more than one architecture and it
> was one of the aim of our work as well but...
>
>> The patch for LoongArch vcpu hotplug link is:
>> https://lore.kernel.org/qemu-devel/[email protected]/T/#t
>
>
> I quickly went through above patches and it looks like this patch-set is mostly
> based on our latest patches which are at [1], [2] and not just at [3], [4]. As I
> could see most of the functions which you have ported to your architecture are
> part of our Qemu repositories [2] which we have yet to push to community. As I
> am working towards RFC V2 patches and which shall be floated soon. It does not
> makes sense for you to duplicate the GED/ACPI changes which are common across
> architectures and which have been derived from the ARM64 vCPU Hotplug support
> original patches.
>
> This will create merge conflicts, will break large part of our original patch-set.
>
> Hence, I would request you to drop patches 1-4 from your patch-set or rebase
> it over ARM64 original patches in a week or 2 week of time. This is to avoid
> spoiling our previous years of hard work for the topic we have been persistently
> making efforts as you can see through the code and our detailed presentations.
Do you have a plan to post new vcpu hotplug patch soon ? If there is, we can
postpone our patch for reviewing and wait for your arm64 vcpu hotplug patch,
and hope that your patch can merge into qemu asap.
We always rebase on the mainline qemu version, rather than personal private tree -:)
Regards
Bibo Mao
>
> Hope you will agree we all need to respect others efforts and time in this
> mode of open-source collaboration.
>
> Rest assured I will help you in review of your architecture specific patch-set
> as it is a work of mutual interest.
>
> Thanks for understanding!
>
>
> Best Wishes,
> Salil.
>
>
>
>> Regards
>> Bibo Mao
>>
>>>
>>>
>>> [3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
>>> https://lore.kernel.org/all/[email protected]/
>>> [4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
>>> https://lore.kernel.org/all/[email protected]/#r
>>>
>>>
>>> In summary, there were some ARM64 Architecture constraints which were being
>>> violated in the earlier patches of the kernel [4] so we had to re-think of the
>>> kernel changes. The Qemu part mostly remains same with some new introductions
>>> of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
>>> in VMM/Qemu.
>>>
>>>
>>> You can follow the KVMForum conference presentations [5], [6] delivered in the
>>> year 2020 and 2023 to get hold of more details related to this.
>>>
>>>
>>> [5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
>>> https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
>>> [6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
>>> https://kvmforum2020.sched.com/event/eE4m
>>>
>>>
>>>
>>>> I ask this question because we propose
>>>> similar patch about LoongArch system in qemu mailing list, and kernel need not be
>>>> modified for vcpu hotplug.
>>>
>>>
>>> Could you please share the link of your patches so that we can have a look and
>>> draw a comparison?
>>>
>>>
>>> Thanks
>>> Salil.
>>>
>>>>
>>>> Regards
>>>> Bibo, mao
>>>>
>>>> 在 2023/7/19 10:35, Salil Mehta 写道:
>>>>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
>>>>>
>>>>> Hello,
>>>>> I have recently started to dabble with ARM CCA stuff and check if our
>>>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
>>>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
>>>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
>>>>> from the host. This might be required to support vCPU Hotplug feature
>>>>> in the realm world in future. vCPU Hotplug is an important feature to
>>>>> support kata-containers in realm world as it reduces the VM boot time
>>>>> and facilitates dynamic adjustment of vCPUs (which I think should be
>>>>> true even with Realm world as current implementation only makes use
>>>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>>>>>
>>>>>
>>>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
>>>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
>>>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
>>>>> PSCI_CPU_ON should undergo similar policy checks and I think,
>>>>>
>>>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
>>>>> 2. This means target REC should be denied by host. Can host call
>>>>> RMI_PSCI_COMPETE in such s case?
>>>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
>>>>> 4. Failure condition (B5.3.3.2) should be amended with
>>>>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
>>>>> post: result == PSCI_DENIED (?)
>>>>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
>>>>> PSCI_CPU_ON flow (D1.4.1)
>>>>>
>>>>>
>>>>> I do understand that ARM CCA support is in its infancy stage and
>>>>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
>>>>> idea right now. But specification changes require lot of time and if
>>>>> this change is really required then it should be further discussed
>>>>> within ARM.
>>>>>
>>>>> Many thanks!
>>>>>
>>>>>
>>>>> Bes regards
>>>>> Salil
>>>>>
>>>>>
>>>>> References:
>>>>>
>>>>> [1] https://developer.arm.com/documentation/den0137/latest/
>>>>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
>>>>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
>
>
Hi Salil
On 25/07/2023 01:05, Salil Mehta wrote:
> Hi Suzuki,
> Sorry for replying late as I was on/off last week to undergo some medical test.
>
>
>> From: Suzuki K Poulose <[email protected]>
>> Sent: Monday, July 24, 2023 5:27 PM
>>
>> Hi Salil
>>
>> On 19/07/2023 10:28, Suzuki K Poulose wrote:
>>> Hi Salil
>>>
>>> Thanks for raising this.
>>>
>>> On 19/07/2023 03:35, Salil Mehta wrote:
>>>> [Reposting it here from Linaro Open Discussion List for more eyes to
>>>> look at]
>>>>
>>>> Hello,
>>>> I have recently started to dabble with ARM CCA stuff and check if our
>>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
>>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
>>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
>>>> from the host. This might be required to support vCPU Hotplug feature
>>>> in the realm world in future. vCPU Hotplug is an important feature to
>>>> support kata-containers in realm world as it reduces the VM boot time
>>>> and facilitates dynamic adjustment of vCPUs (which I think should be
>>>> true even with Realm world as current implementation only makes use
>>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>>>>
>>>>
>>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
>>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
>>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
>>>> PSCI_CPU_ON should undergo similar policy checks and I think,
>>>>
>>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
>>>> 2. This means target REC should be denied by host. Can host call
>>>> RMI_PSCI_COMPETE in such s case?
>>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
>>>
>>> The Realm exit with EXIT_PSCI already provides the parameters passed
>>> onto the PSCI request. This happens for all PSCI calls except
>>> (PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
>>> the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses
>>> the request (wherever applicable).
>>>
>>> However, the RMM spec currently doesn't allow denying the request.
>>> i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
>>> We will address this in the RMM spec and get back to you.
>>
>> This is now resolved in RMMv1.0-eac3 spec, available here [0].
>>
>> This allows the host to DENY a PSCI_CPU_ON request. The RMM ensures that
>> the response doesn't violate the security guarantees by checking the
>> state of the target REC.
>>
>> [0] https://developer.arm.com/documentation/den0137/latest/
>
>
> Many thanks for taking this up proactively and getting it done as well
> very efficiently. Really appreciate this!
>
> I acknowledge below new changes part of the newly released RMM
> Specification [3] (Page-2) (Release Information 1.0-eac3 20-07-2023):
>
> 1. Addition of B2.19 PsciReturnCodePermitted function [3] (Page-126)
> 2. Addition of 'status' in B3.3.7.2 Failure conditions of the
> B3.3.7 RMI_PSCI_COMPLETE command [3] (Page-160)
>
>
> Some Further Suggestions:
> 1. It would be really helpful if PSCI_DENIED can be accommodated somewhere
> in the flow diagram (D1.4.1 PSCI_CPU_ON flow) [3] (Page-297) as well.
Good point, yes, will get that added.
> 2. You would need changes to handle the return value of the PSCI_DENIED
> in this below patch [2] as well from ARM CCA series [1]
>
Of course. Please note that the series [1] is based on RMMv1.0-beta0 and
we are in the process of rebasing our changes to v1.0-eac3, which
includes a lot of other changes. The updated series will have all the
required changes.
Kind regards
Suzuki
>
> @James, Any further thoughts on this?
>
>
> References:
> [1] [RFC PATCH 00/28] arm64: Support for Arm CCA in KVM
> [2] [RFC PATCH 19/28] KVM: arm64: Validate register access for a Realm VM
> https://lore.kernel.org/lkml/[email protected]/T/#m6c10b9a27c4a724967c1800facacaa9443b38b4c
> [3] ARM Realm Management Monitor specification(DEN0137 1.0-eac3)
> https://developer.arm.com/documentation/den0137/latest/
>
>
> Thanks
> Salil.
>
>
>>>> 4. Failure condition (B5.3.3.2) should be amended with
>>>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
>>>> post: result == PSCI_DENIED (?)
>>>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
>>>> PSCI_CPU_ON flow (D1.4.1)
>>>>
>>>> I do understand that ARM CCA support is in its infancy stage and
>>>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
>>>> idea right now. But specification changes require lot of time and if
>>>> this change is really required then it should be further discussed
>>>> within ARM.
>>>>
>>>> Many thanks!
>>>>
>>>>
>>>> Bes regards
>>>> Salil
>>>>
>>>>
>>>> References:
>>>>
>>>> [1] https://developer.arm.com/documentation/den0137/latest/
>>>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
>>>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
>
Hi Bibo,
> From: bibo mao <[email protected]>
> Sent: Tuesday, July 25, 2023 7:24 AM
> To: Salil Mehta <[email protected]>
[...]
> 在 2023/7/25 13:45, Salil Mehta 写道:
> > Hello,
> >
> >> From: bibo mao <[email protected]>
> >> Sent: Tuesday, July 25, 2023 2:14 AM
> >> To: Salil Mehta <[email protected]>
[...]
> >> 在 2023/7/25 08:56, Salil Mehta 写道:
> >>>
> >>>> From: bibo mao <[email protected]>
> >>>> Sent: Tuesday, July 25, 2023 1:29 AM
> >>>> To: Salil Mehta <[email protected]>
> >>>> Cc: Catalin Marinas <[email protected]>; Jonathan Cameron
> >>>> <[email protected]>; Marc Zyngier <[email protected]>; Will Deacon
> >>>> <[email protected]>; [email protected]; [email protected];
> >>>> [email protected]; [email protected]; Salil Mehta
> >>>> <[email protected]>; [email protected]; yuzenghui
> >>>> <[email protected]>; [email protected]; linux-arm-
> >>>> [email protected]; [email protected];
> >>>> [email protected]; [email protected]; [email protected];
> >>>> [email protected]; Suzuki K Poulose <[email protected]>; Jean-
> >>>> Philippe Brucker <[email protected]>; [email protected]; [email protected]
> >>>> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?
> >>>>
> >>>> Is vcpu hotplug supported in arm virt-machine now?
> >>>
> >>> Not yet. We are working on it. Please check the RFCs being tested.
> >>>
> >>>
> >>> [1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
> >>> https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>> [2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
> >>> https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
> >>>
> >>>
> >>>> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
> >>>> I do not know why it is not merged.
> >>>
> >>>
> >>> I think you are referring to patches [3], [4]? Please follow the discussion
> >>> for details.
> >>
> >> yeap, we reference the patch [3], [4] and benefit from them greatly -:)
> >
> >
> > I am glad that our current work is useful to more than one architecture and it
> > was one of the aim of our work as well but...
> >
> >> The patch for LoongArch vcpu hotplug link is:
> >> https://lore.kernel.org/qemu-devel/[email protected]/T/#t
> >
> >
> > I quickly went through above patches and it looks like this patch-set is mostly
> > based on our latest patches which are at [1], [2] and not just at [3], [4]. As I
> > could see most of the functions which you have ported to your architecture are
> > part of our Qemu repositories [2] which we have yet to push to community. As I
> > am working towards RFC V2 patches and which shall be floated soon. It does not
> > makes sense for you to duplicate the GED/ACPI changes which are common across
> > architectures and which have been derived from the ARM64 vCPU Hotplug support
> > original patches.
> >
> > This will create merge conflicts, will break large part of our original patch-set.
> >
> > Hence, I would request you to drop patches 1-4 from your patch-set or rebase
> > it over ARM64 original patches in a week or 2 week of time. This is to avoid
> > spoiling our previous years of hard work for the topic we have been persistently
> > making efforts as you can see through the code and our detailed presentations.
>
> Do you have a plan to post new vcpu hotplug patch soon ? If there is, we can
> postpone our patch for reviewing and wait for your arm64 vcpu Hotplug patch,
> and hope that your patch can merge into qemu asap.
Yes, we are working on it and plan to post it soon in 2 weeks of time.
>
> We always rebase on the mainline qemu version, rather than personal private tree -:)
Ok. I thought if you could take code from private branch then technically you
could rebase from it as well :)
Anyways, I will be soon posting the changes and it will have those ACPI/GED
changes as well which we can try to get accepted quickly. Many thanks!
Best wishes,
Salil
> Regards
> Bibo Mao
> >
> > Hope you will agree we all need to respect others efforts and time in this
> > mode of open-source collaboration.
> >
> > Rest assured I will help you in review of your architecture specific patch-set
> > as it is a work of mutual interest.
> >
> > Thanks for understanding!
> >
> >
> > Best Wishes,
> > Salil.
> >
> >> Regards
> >> Bibo Mao
> >>
> >>>
> >>>
> >>> [3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
> >>> https://lore.kernel.org/all/[email protected]/
> >>> [4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
> >>> https://lore.kernel.org/all/[email protected]/#r
> >>>
> >>>
> >>> In summary, there were some ARM64 Architecture constraints which were being
> >>> violated in the earlier patches of the kernel [4] so we had to re-think of the
> >>> kernel changes. The Qemu part mostly remains same with some new introductions
> >>> of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
> >>> in VMM/Qemu.
> >>>
> >>>
> >>> You can follow the KVMForum conference presentations [5], [6] delivered in the
> >>> year 2020 and 2023 to get hold of more details related to this.
> >>>
> >>>
> >>> [5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
> >>> https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
> >>> [6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
> >>> https://kvmforum2020.sched.com/event/eE4m
> >>>
> >>>
> >>>> I ask this question because we propose
> >>>> similar patch about LoongArch system in qemu mailing list, and kernel need not be
> >>>> modified for vcpu hotplug.
> >>>
> >>>
> >>> Could you please share the link of your patches so that we can have a look and
> >>> draw a comparison?
> >>>
> >>>
> >>> Thanks
> >>> Salil.
> >>>
> >>>>
> >>>> Regards
> >>>> Bibo, mao
> >>>>
> >>>> 在 2023/7/19 10:35, Salil Mehta 写道:
> >>>>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
> >>>>>
> >>>>> Hello,
> >>>>> I have recently started to dabble with ARM CCA stuff and check if our
> >>>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> >>>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> >>>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> >>>>> from the host. This might be required to support vCPU Hotplug feature
> >>>>> in the realm world in future. vCPU Hotplug is an important feature to
> >>>>> support kata-containers in realm world as it reduces the VM boot time
> >>>>> and facilitates dynamic adjustment of vCPUs (which I think should be
> >>>>> true even with Realm world as current implementation only makes use
> >>>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >>>>>
> >>>>>
> >>>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
> >>>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> >>>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> >>>>> PSCI_CPU_ON should undergo similar policy checks and I think,
> >>>>>
> >>>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> >>>>> 2. This means target REC should be denied by host. Can host call
> >>>>> RMI_PSCI_COMPETE in such s case?
> >>>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> >>>>> 4. Failure condition (B5.3.3.2) should be amended with
> >>>>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >>>>> post: result == PSCI_DENIED (?)
> >>>>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >>>>> PSCI_CPU_ON flow (D1.4.1)
> >>>>>
> >>>>>
> >>>>> I do understand that ARM CCA support is in its infancy stage and
> >>>>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> >>>>> idea right now. But specification changes require lot of time and if
> >>>>> this change is really required then it should be further discussed
> >>>>> within ARM.
> >>>>>
> >>>>> Many thanks!
> >>>>>
> >>>>>
> >>>>> Bes regards
> >>>>> Salil
> >>>>>
> >>>>>
> >>>>> References:
> >>>>>
> >>>>> [1] https://developer.arm.com/documentation/den0137/latest/
> >>>>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>>>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
Hi Suzuki,
> From: Suzuki K Poulose <[email protected]>
> Sent: Tuesday, July 25, 2023 12:20 PM
>
> Hi Salil
>
> On 25/07/2023 01:05, Salil Mehta wrote:
> > Hi Suzuki,
> > Sorry for replying late as I was on/off last week to undergo some medical test.
> >
> >> From: Suzuki K Poulose <[email protected]>
> >> Sent: Monday, July 24, 2023 5:27 PM
> >>
> >> Hi Salil
> >>
> >> On 19/07/2023 10:28, Suzuki K Poulose wrote:
> >>> Hi Salil
> >>>
> >>> Thanks for raising this.
> >>>
> >>> On 19/07/2023 03:35, Salil Mehta wrote:
> >>>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
> >>>>
> >>>> Hello,
> >>>> I have recently started to dabble with ARM CCA stuff and check if our
> >>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> >>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> >>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> >>>> from the host. This might be required to support vCPU Hotplug feature
> >>>> in the realm world in future. vCPU Hotplug is an important feature to
> >>>> support kata-containers in realm world as it reduces the VM boot time
> >>>> and facilitates dynamic adjustment of vCPUs (which I think should be
> >>>> true even with Realm world as current implementation only makes use
> >>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >>>>
> >>>>
> >>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
> >>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> >>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> >>>> PSCI_CPU_ON should undergo similar policy checks and I think,
> >>>>
> >>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> >>>> 2. This means target REC should be denied by host. Can host call
> >>>> RMI_PSCI_COMPETE in such s case?
> >>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> >>>
> >>> The Realm exit with EXIT_PSCI already provides the parameters passed
> >>> onto the PSCI request. This happens for all PSCI calls except
> >>> (PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
> >>> the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses
> >>> the request (wherever applicable).
> >>>
> >>> However, the RMM spec currently doesn't allow denying the request.
> >>> i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
> >>> We will address this in the RMM spec and get back to you.
> >>
> >> This is now resolved in RMMv1.0-eac3 spec, available here [0].
> >>
> >> This allows the host to DENY a PSCI_CPU_ON request. The RMM ensures that
> >> the response doesn't violate the security guarantees by checking the
> >> state of the target REC.
> >>
> >> [0] https://developer.arm.com/documentation/den0137/latest/
> >
> >
> > Many thanks for taking this up proactively and getting it done as well
> > very efficiently. Really appreciate this!
> >
> > I acknowledge below new changes part of the newly released RMM
> > Specification [3] (Page-2) (Release Information 1.0-eac3 20-07-2023):
> >
> > 1. Addition of B2.19 PsciReturnCodePermitted function [3] (Page-126)
> > 2. Addition of 'status' in B3.3.7.2 Failure conditions of the
> > B3.3.7 RMI_PSCI_COMPLETE command [3] (Page-160)
> >
> >
> > Some Further Suggestions:
> > 1. It would be really helpful if PSCI_DENIED can be accommodated somewhere
> > in the flow diagram (D1.4.1 PSCI_CPU_ON flow) [3] (Page-297) as well.
>
> Good point, yes, will get that added.
Great. Thanks!
> > 2. You would need changes to handle the return value of the PSCI_DENIED
> > in this below patch [2] as well from ARM CCA series [1]
> >
>
> Of course. Please note that the series [1] is based on RMMv1.0-beta0 and
> we are in the process of rebasing our changes to v1.0-eac3, which
> includes a lot of other changes. The updated series will have all the
> required changes.
Ok. When are you planning to post this new series with v1.0-eac3 changes?
Thanks
Salil.
> Kind regards
> Suzuki
>
> > @James, Any further thoughts on this?
> >
> >
> > References:
> > [1] [RFC PATCH 00/28] arm64: Support for Arm CCA in KVM
> > [2] [RFC PATCH 19/28] KVM: arm64: Validate register access for a Realm VM
> > https://lore.kernel.org/lkml/[email protected]/T/#m6c10b9a27c4a724967c1800facacaa9443b38b4c
> > [3] ARM Realm Management Monitor specification(DEN0137 1.0-eac3)
> > https://developer.arm.com/documentation/den0137/latest/
> >
> >
> > Thanks
> > Salil.
> >
> >
> >>>> 4. Failure condition (B5.3.3.2) should be amended with
> >>>> runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >>>> post: result == PSCI_DENIED (?)
> >>>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >>>> PSCI_CPU_ON flow (D1.4.1)
> >>>>
> >>>> I do understand that ARM CCA support is in its infancy stage and
> >>>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> >>>> idea right now. But specification changes require lot of time and if
> >>>> this change is really required then it should be further discussed
> >>>> within ARM.
> >>>>
> >>>> Many thanks!
> >>>>
> >>>>
> >>>> Bes regards
> >>>> Salil
> >>>>
> >>>> References:
> >>>>
> >>>> [1] https://developer.arm.com/documentation/den0137/latest/
> >>>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
On 27/07/2023 15:24, Salil Mehta wrote:
> Hi Suzuki,
>
>> From: Suzuki K Poulose <[email protected]>
>> Sent: Tuesday, July 25, 2023 12:20 PM
>>
>> Hi Salil
>>
>> On 25/07/2023 01:05, Salil Mehta wrote:
>>> Hi Suzuki,
>>> Sorry for replying late as I was on/off last week to undergo some medical test.
>>>
>>>> From: Suzuki K Poulose <[email protected]>
>>>> Sent: Monday, July 24, 2023 5:27 PM
>>>>
>>>> Hi Salil
>>>>
>>>> On 19/07/2023 10:28, Suzuki K Poulose wrote:
>>>>> Hi Salil
>>>>>
>>>>> Thanks for raising this.
>>>>>
>>>>> On 19/07/2023 03:35, Salil Mehta wrote:
>>>>>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
>>>>>>
>>>>>> Hello,
>>>>>> I have recently started to dabble with ARM CCA stuff and check if our
>>>>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
>>>>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
>>>>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
>>>>>> from the host. This might be required to support vCPU Hotplug feature
>>>>>> in the realm world in future. vCPU Hotplug is an important feature to
>>>>>> support kata-containers in realm world as it reduces the VM boot time
>>>>>> and facilitates dynamic adjustment of vCPUs (which I think should be
>>>>>> true even with Realm world as current implementation only makes use
>>>>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
>>>>>>
>>>>>>
>>>>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
>>>>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
>>>>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
>>>>>> PSCI_CPU_ON should undergo similar policy checks and I think,
>>>>>>
>>>>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
>>>>>> 2. This means target REC should be denied by host. Can host call
>>>>>> RMI_PSCI_COMPETE in such s case?
>>>>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
>>>>>
>>>>> The Realm exit with EXIT_PSCI already provides the parameters passed
>>>>> onto the PSCI request. This happens for all PSCI calls except
>>>>> (PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
>>>>> the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses
>>>>> the request (wherever applicable).
>>>>>
>>>>> However, the RMM spec currently doesn't allow denying the request.
>>>>> i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
>>>>> We will address this in the RMM spec and get back to you.
>>>>
>>>> This is now resolved in RMMv1.0-eac3 spec, available here [0].
>>>>
>>>> This allows the host to DENY a PSCI_CPU_ON request. The RMM ensures that
>>>> the response doesn't violate the security guarantees by checking the
>>>> state of the target REC.
>>>>
>>>> [0] https://developer.arm.com/documentation/den0137/latest/
>>>
>>>
>>> Many thanks for taking this up proactively and getting it done as well
>>> very efficiently. Really appreciate this!
>>>
>>> I acknowledge below new changes part of the newly released RMM
>>> Specification [3] (Page-2) (Release Information 1.0-eac3 20-07-2023):
>>>
>>> 1. Addition of B2.19 PsciReturnCodePermitted function [3] (Page-126)
>>> 2. Addition of 'status' in B3.3.7.2 Failure conditions of the
>>> B3.3.7 RMI_PSCI_COMPLETE command [3] (Page-160)
>>>
>>>
>>> Some Further Suggestions:
>>> 1. It would be really helpful if PSCI_DENIED can be accommodated somewhere
>>> in the flow diagram (D1.4.1 PSCI_CPU_ON flow) [3] (Page-297) as well.
>>
>> Good point, yes, will get that added.
>
>
> Great. Thanks!
>
>
>>> 2. You would need changes to handle the return value of the PSCI_DENIED
>>> in this below patch [2] as well from ARM CCA series [1]
>>>
>>
>> Of course. Please note that the series [1] is based on RMMv1.0-beta0 and
>> we are in the process of rebasing our changes to v1.0-eac3, which
>> includes a lot of other changes. The updated series will have all the
>> required changes.
>
>
> Ok. When are you planning to post this new series with v1.0-eac3 changes?
Please see :
https://lkml.kernel.org/r/[email protected]
Suzuki