Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <7a1fa398-3304-cb30-10c0-83e12ac9e8ac@linux.ibm.com>
Date:   Mon, 24 Jan 2022 10:28:15 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.4.0
Subject: Re: [PATCH v2 17/30] KVM: s390: mechanism to enable guest zPCI
 Interpretation
Content-Language: en-US
To:     Pierre Morel <pmorel@linux.ibm.com>, linux-s390@vger.kernel.org
Cc:     alex.williamson@redhat.com, cohuck@redhat.com,
        schnelle@linux.ibm.com, farman@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com,
        vneethv@linux.ibm.com, oberpar@linux.ibm.com, freude@linux.ibm.com,
        thuth@redhat.com, pasic@linux.ibm.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20220114203145.242984-1-mjrosato@linux.ibm.com>
 <20220114203145.242984-18-mjrosato@linux.ibm.com>
 <7125d611-5440-09ae-429a-7a087dd77868@linux.ibm.com>
From:   Matthew Rosato <mjrosato@linux.ibm.com>
In-Reply-To: <7125d611-5440-09ae-429a-7a087dd77868@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 1/24/22 9:24 AM, Pierre Morel wrote:
> 
> 
> On 1/14/22 21:31, Matthew Rosato wrote:
>> The guest must have access to certain facilities in order to allow
>> interpretive execution of zPCI instructions and adapter event
>> notifications.  However, there are some cases where a guest might
>> disable interpretation -- provide a mechanism via which we can defer
>> enabling the associated zPCI interpretation facilities until the guest
>> indicates it wishes to use them.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---
>>   arch/s390/include/asm/kvm_host.h |  4 ++++
>>   arch/s390/kvm/kvm-s390.c         | 40 ++++++++++++++++++++++++++++++++
>>   arch/s390/kvm/kvm-s390.h         | 10 ++++++++
>>   3 files changed, 54 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h 
>> b/arch/s390/include/asm/kvm_host.h
>> index 3f147b8d050b..38982c1de413 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -252,7 +252,10 @@ struct kvm_s390_sie_block {
>>   #define ECB2_IEP    0x20
>>   #define ECB2_PFMFI    0x08
>>   #define ECB2_ESCA    0x04
>> +#define ECB2_ZPCI_LSI    0x02
>>       __u8    ecb2;                   /* 0x0062 */
>> +#define ECB3_AISI    0x20
>> +#define ECB3_AISII    0x10
>>   #define ECB3_DEA 0x08
>>   #define ECB3_AES 0x04
>>   #define ECB3_RI  0x01
>> @@ -938,6 +941,7 @@ struct kvm_arch{
>>       int use_cmma;
>>       int use_pfmfi;
>>       int use_skf;
>> +    int use_zpci_interp;
>>       int user_cpu_state_ctrl;
>>       int user_sigp;
>>       int user_stsi;
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index ab8b56deed11..b6c32fc3b272 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -1029,6 +1029,44 @@ static int kvm_s390_vm_set_crypto(struct kvm 
>> *kvm, struct kvm_device_attr *attr)
>>       return 0;
>>   }
>> +static void kvm_s390_vcpu_pci_setup(struct kvm_vcpu *vcpu)
>> +{
>> +    /* Only set the ECB bits after guest requests zPCI interpretation */
>> +    if (!vcpu->kvm->arch.use_zpci_interp)
>> +        return;
>> +
>> +    vcpu->arch.sie_block->ecb2 |= ECB2_ZPCI_LSI;
>> +    vcpu->arch.sie_block->ecb3 |= ECB3_AISII + ECB3_AISI;
> 
> As far as I understood, the interpretation is only possible if a gisa 
> designation is associated with the PCI function via CLP enable.
> 

This is true.  Once ECB is enabled, you must have either a SHM bit on 
for emulated device support or SHM bits off + a GISA designation 
registered for interpretation.  Otherwise, PCI instructions will fail.

> Why do we setup the SIE ECB only when the guest requests for 
> interpretation and not systematically in vcpu_setup?

Once the ECB is enabled for a guest, emulated device FHs must have a SHM 
bit in order to continue working properly (so do passthrough devices 
that don't setup interpretation).  This was not a requirement before 
this series -- simply having the ECB bit off would ensure intercepts for 
all devices regardless of SHM bit settings, so by doing an opt-in once 
the guest indicates it will be doing interpretation we can preserve 
backwards-compatibility with an initial mode where SHM bits are not 
necessarily required.  However once userspace indicates it understands 
interpretation, we can assume it is will also use SHM bits properly.

> 
> If ECB2_ZPCI_LSI, ECB3_AISII or ECB3_AISI have an effect when the gisa 
> designation is not specified shouldn't we have a way to clear these bits?
> 

I'm not sure that's necessary -- The idea here was for the userspace to 
indicate 1) that it knows how to setup for interpreted devices and 2) 
that it has a guest that wants to use at least 1 interpreted device.
Once we know that userspace understands how to manage interpreted 
devices (implied by its use of these new vfio feature ioctls) I think it 
should be OK to leave these bits on and expect userspace to always do 
the appropriate steps (SHM bits for emulated devices / forced intercept 
passthrough devices, GISA designation for interpreted devices).