From:   Cornelia Huck <cohuck@redhat.com>
To:     Suzuki K Poulose <suzuki.poulose@arm.com>,
        Andrew Jones <andrew.jones@linux.dev>,
        Jean-Philippe Brucker <jpb@kernel.org>
Cc:     Jean-Philippe Brucker <jean-philippe@linaro.org>,
        Itaru Kitayama <itaru.kitayama@gmail.com>,
        linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org,
        kvm@vger.kernel.org, kvmarm@lists.linux.dev,
        linux-arm-kernel@lists.infradead.org,
        Alexandru Elisei <alexandru.elisei@arm.com>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Chao Peng <chao.p.peng@linux.intel.com>,
        Christoffer Dall <christoffer.dall@arm.com>,
        Fuad Tabba <tabba@google.com>,
        James Morse <james.morse@arm.com>,
        Joey Gouly <Joey.Gouly@arm.com>, Marc Zyngier <maz@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Oliver Upton <oliver.upton@linux.dev>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Quentin Perret <qperret@google.com>,
        Sean Christopherson <seanjc@google.com>,
        Steven Price <steven.price@arm.com>,
        Thomas Huth <thuth@redhat.com>, Will Deacon <will@kernel.org>,
        Zenghui Yu <yuzenghui@huawei.com>, kvmarm@lists.cs.columbia.edu
Subject: Re: [RFC] Support for Arm CCA VMs on Linux
In-Reply-To: <2418536c-2658-18d6-f70c-c1af5adaa816@arm.com>
Organization: Red Hat GmbH
References: <20230127112248.136810-1-suzuki.poulose@arm.com>
 <Y9PtKJ3Wicc19JF1@myrica>
 <CANW9uyud8RTkqgiL=64wV712QMxtAyubqeyCJ0vpcADJ42VqJA@mail.gmail.com>
 <Y/8Y3WLmiw6+Z5AS@myrica>
 <CANW9uysnvGCwANu+_6dp9+3rvHGOkThT9d0K2qpQV4exdmYWoA@mail.gmail.com>
 <20230303094618.GC361458@myrica>
 <1c91b777-982e-e71a-4829-51744e9555c5@arm.com>
 <20230303113905.GD361458@myrica> <20230303120800.ahtyc6et77ig4s27@orel>
 <2418536c-2658-18d6-f70c-c1af5adaa816@arm.com>
User-Agent: Notmuch/0.37 (https://notmuchmail.org)
Date:   Fri, 03 Mar 2023 14:06:07 +0100
Message-ID: <875ybi0ytc.fsf@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

On Fri, Mar 03 2023, Suzuki K Poulose <suzuki.poulose@arm.com> wrote:

> On 03/03/2023 12:08, Andrew Jones wrote:
>> On Fri, Mar 03, 2023 at 11:39:05AM +0000, Jean-Philippe Brucker wrote:
>>> On Fri, Mar 03, 2023 at 09:54:47AM +0000, Suzuki K Poulose wrote:
>>>> On 03/03/2023 09:46, Jean-Philippe Brucker wrote:
>>>>> On Thu, Mar 02, 2023 at 07:12:24AM +0900, Itaru Kitayama wrote:
>>>>>>>> I've tried your series in Real on CCA Host, but the KVM arch init
>>>>>>>> emits an Invalid argument error and terminates.
>>>>>
>>>>> This was the KVM_SET_ONE_REG for the SVE vector size. During my tests I
>>>>> didn't enable SVE in the host but shrinkwrap enables more options.
>>>>
>>>> Does the Qemu check for SVE capability on /dev/kvm ? For kvmtool, we
>>>> changed to using the VM instance and that would prevent using SVE,
>>>> until the RMM supports it.
>>>
>>> Yes, QEMU does check the SVE cap on /dev/kvm. I can propose changing it or
>>> complementing it with a VM check in my next version, it seems to work
>>> (though I need to double-check the VM fd lifetime). Same goes for
>>> KVM_CAP_STEAL_TIME, which I need to disable explicitly at the moment.
>> 
>> I'm probably missing something since I haven't looked at this, but I'm
>> wondering what the "VM instance" check is and why it should be necessary.
>
> Userspace can check for a KVM_CAP_ on KVM fd (/dev/kvm) or a VM fd
> (returned via KVM_CREATE_VM).
>
>> Shouldn't KVM only expose capabilities which it can provide? I.e. the
>
> Correct, given now that we have different "types" of VMs possible on
> Arm64, (Normal vs Realm vs pVM), the capabilities of each of these
> could be different and thus we should use the KVM_CAP_ on the VM fd (
> referred to VM instance above) and not the generic KVM fd.

Using the vm ioctl is even encouraged in the doc for
KVM_CHECK_EXTENSION:

"Based on their initialization different VMs may have different capabilities.
It is thus encouraged to use the vm ioctl to query for capabilities"

It would probably make sense to convert QEMU to use the vm ioctl
everywhere (the wrapper falls back to the global version on failure
anyway.)

>
>> "VM instance" check should be done by KVM and, when it fails, the SVE and
>> steal-time capabilities should return 0.
>> 
>
> Correct.
>
> Suzuki
>
>> Thanks,
>> drew