Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7da93c6e-1cbf-8840-282e-f115197b80c4@arm.com>
Date:   Wed, 19 Jul 2023 10:28:29 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.12.0
Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world
 might require ARM spec change?
To:     Salil Mehta <salil.mehta@huawei.com>,
        "steven.price@arm.com" <steven.price@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.org>,
        "james.morse@arm.com" <james.morse@arm.com>,
        "linux-coco@lists.linux.dev" <linux-coco@lists.linux.dev>,
        "kvmarm@lists.linux.dev" <kvmarm@lists.linux.dev>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        Marc Zyngier <maz@kernel.org>, Will Deacon <will@kernel.org>,
        "christoffer.dall@arm.com" <christoffer.dall@arm.com>,
        "oliver.upton@linux.dev" <oliver.upton@linux.dev>,
        "mark.rutland@arm.com" <mark.rutland@arm.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        Salil Mehta <salil.mehta@opnsrc.net>,
        "andrew.jones@linux.dev" <andrew.jones@linux.dev>,
        yuzenghui <yuzenghui@huawei.com>,
        "kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Gareth Stockwell <Gareth.Stockwell@arm.com>
References: <9cb24131a09a48e9a622e92bf8346c9d@huawei.com>
From:   Suzuki K Poulose <suzuki.poulose@arm.com>
In-Reply-To: <9cb24131a09a48e9a622e92bf8346c9d@huawei.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Salil

Thanks for raising this.

On 19/07/2023 03:35, Salil Mehta wrote:
> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
> 
> Hello,
> I have recently started to dabble with ARM CCA stuff and check if our
> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> from the host. This might be required to support vCPU Hotplug feature
> in the realm world in future. vCPU Hotplug is an important feature to
> support kata-containers in realm world as it reduces the VM boot time
> and facilitates dynamic adjustment of vCPUs (which I think should be
> true even with Realm world as current implementation only makes use
> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> 
> 
> As per our recent changes [2], [3] related to support vCPU Hotplug on
> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> PSCI_CPU_ON should undergo similar policy checks and I think,
> 
> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> 2. This means target REC should be denied by host. Can host call
>     RMI_PSCI_COMPETE in such s case?
> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED

The Realm exit with EXIT_PSCI already provides the parameters passed
onto the PSCI request. This happens for all PSCI calls except
(PSCI_VERSION and PSCI_FEAUTRES). The hyp could forward these exits to
the VMM and could invoke the RMI_PSCI_COMPLETE only when the VMM blesses 
the request (wherever applicable).

However, the RMM spec currently doesn't allow denying the request.
i.e., without RMI_PSCI_COMPLETE, the REC cannot be scheduled back in.
We will address this in the RMM spec and get back to you.

Kind regards
Suzuki


> 4. Failure condition (B5.3.3.2) should be amended with
>     runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
>              post: result == PSCI_DENIED (?)
> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
>     PSCI_CPU_ON flow (D1.4.1)
>    
> 
> I do understand that ARM CCA support is in its infancy stage and
> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> idea right now. But specification changes require lot of time and if
> this change is really required then it should be further discussed
> within ARM.
> 
> Many thanks!
> 
> 
> Bes regards
> Salil
> 
> 
> References:
> 
> [1] https://developer.arm.com/documentation/den0137/latest/
> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
>