Received: by 10.213.65.68 with SMTP id h4csp229276imn; Fri, 16 Mar 2018 00:52:56 -0700 (PDT) X-Google-Smtp-Source: AG47ELvIVMzSENCS/bLCFCVyNQqks3KgAKmAu8rXjLCtrArABRcQSXcApthCWiPqsaFenw/uvZnd X-Received: by 2002:a17:902:1a4:: with SMTP id b33-v6mr1047677plb.321.1521186776269; Fri, 16 Mar 2018 00:52:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1521186776; cv=none; d=google.com; s=arc-20160816; b=aa5ny/K0ZLdcrpmWBEUGNQO9LBP5MKn4i6J3/aJZl84nPsgSD37X0+Aqt2DCXSGMqH f6EUDcYomwaEmzWJAzqrs99qlU2Do6bSukFkBms0t5Rd5HUuahuhyXh9ndeDMYKistVn u2fteO+dcN9wIS1EMud/UWgVSW/8A+t18u/CrO9+gqm6E4b+Hgu9MA4ZSYRJTEm1y8Hk suB3vAJp6NvIVggfzTLv/N3D8UK3MKQhPBWk0mFai7gqeF9sWzUXdMDfKWNIO9t9n/+c 1ZuDyFcvbGA83V9Qo2ECo3qyWBEZv4YaqC5YRKJ+E3I/k3eVW6JyavV81tapztEs8GCE D3/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:content-language :content-transfer-encoding:in-reply-to:mime-version:user-agent:date :from:references:cc:to:subject:arc-authentication-results; bh=rNCiUWCiFRiS18JZpu/Iw8GxhSOSeeFKTPRITz5tnG4=; b=dUdwA7Ks64O5mC9vGfrMBB8qW3sKEOFQaDECm6n8fryRLzaufCiCvW82r8P1IMznnX 4sCrOANSfiNvZLhJNwGjz2qbDOvsgTZLPuKRlcZLKvfWIpDwsgA9SlXpCnm4CWkxSQJs kWMsa+LZKqw7d31dNeTtePu+SMjnzL8h/u81C1XE3m9U0Lomf3gaFywOekxRiDEalfLF OFWtjakxJ1Qq/tdWmKq1dANedfyaVJDhj6nzOIf+ifPgYs5RZmBLMMmLVBQVFPxbgChX qOPOFETOFRnWBpoIMUbMkmvcLUmukiJB9II9yntA6GmIlVNAWHa3oR+MSf380sUWaknT FXRg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g24si4653891pgv.396.2018.03.16.00.52.41; Fri, 16 Mar 2018 00:52:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753330AbeCPHvv (ORCPT + 99 others); Fri, 16 Mar 2018 03:51:51 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:57426 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752014AbeCPHvs (ORCPT ); Fri, 16 Mar 2018 03:51:48 -0400 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w2G7nRCt089336 for ; Fri, 16 Mar 2018 03:51:48 -0400 Received: from e06smtp12.uk.ibm.com (e06smtp12.uk.ibm.com [195.75.94.108]) by mx0b-001b2d01.pphosted.com with ESMTP id 2gr9fkgw0b-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Fri, 16 Mar 2018 03:51:47 -0400 Received: from localhost by e06smtp12.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 16 Mar 2018 07:51:46 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp12.uk.ibm.com (192.168.101.142) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 16 Mar 2018 07:51:41 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w2G7pff157475200; Fri, 16 Mar 2018 07:51:41 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 290DE42042; Fri, 16 Mar 2018 07:43:54 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7A20B42041; Fri, 16 Mar 2018 07:43:53 +0000 (GMT) Received: from [9.152.224.146] (unknown [9.152.224.146]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 16 Mar 2018 07:43:53 +0000 (GMT) Subject: Re: [PATCH v3 04/14] KVM: s390: device attribute to set AP interpretive execution To: Tony Krowiak , Halil Pasic , linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@de.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, kwankhede@nvidia.com, bjsdjshi@linux.vnet.ibm.com, pbonzini@redhat.com, alex.williamson@redhat.com, alifm@linux.vnet.ibm.com, mjrosato@linux.vnet.ibm.com, jjherne@linux.vnet.ibm.com, thuth@redhat.com, berrange@redhat.com, fiuczy@linux.vnet.ibm.com, buendgen@de.ibm.com References: <1521051954-25715-1-git-send-email-akrowiak@linux.vnet.ibm.com> <1521051954-25715-5-git-send-email-akrowiak@linux.vnet.ibm.com> <21bd029b-3500-3461-ce98-68ad3ae9b647@linux.vnet.ibm.com> <46a7e838-2be2-9587-6eb2-3bba95485609@linux.vnet.ibm.com> <5ed8017b-0168-9a50-234b-cfe9258eab72@linux.vnet.ibm.com> <17683324-f6e4-4328-54c1-1fce572faecd@linux.vnet.ibm.com> <8e10f1cb-3722-d231-2603-b7867420ac0a@linux.vnet.ibm.com> <5dd1bcd3-5d17-37c1-1184-7f75a1fd32bc@linux.vnet.ibm.com> From: Pierre Morel Date: Fri, 16 Mar 2018 08:51:40 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <5dd1bcd3-5d17-37c1-1184-7f75a1fd32bc@linux.vnet.ibm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 18031607-0008-0000-0000-000004DED118 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18031607-0009-0000-0000-00001E71DE32 Message-Id: <68e9e3ea-f99a-da88-5e56-21e38b438b4f@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-03-16_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1803160096 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 16/03/2018 00:39, Tony Krowiak wrote: > On 03/15/2018 01:56 PM, Pierre Morel wrote: >> On 15/03/2018 18:21, Tony Krowiak wrote: >>> On 03/15/2018 11:45 AM, Pierre Morel wrote: >>>> On 15/03/2018 16:26, Tony Krowiak wrote: >>>>> On 03/15/2018 09:00 AM, Pierre Morel wrote: >>>>>> On 14/03/2018 22:57, Halil Pasic wrote: >>>>>>> >>>>>>> On 03/14/2018 07:25 PM, Tony Krowiak wrote: >>>>>>>> The VFIO AP device model exploits interpretive execution of AP >>>>>>>> instructions (APIE) to provide guests passthrough access to AP >>>>>>>> devices. This patch introduces a new device attribute in the >>>>>>>> KVM_S390_VM_CRYPTO device attribute group to set APIE from >>>>>>>> the VFIO AP device defined on the guest. >>>>>>>> >>>>>>>> Signed-off-by: Tony Krowiak >>>>>>>> --- >>>>>>> [..] >>>>>>> >>>>>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c >>>>>>>> index a60c45b..bc46b67 100644 >>>>>>>> --- a/arch/s390/kvm/kvm-s390.c >>>>>>>> +++ b/arch/s390/kvm/kvm-s390.c >>>>>>>> @@ -815,6 +815,19 @@ static int kvm_s390_vm_set_crypto(struct >>>>>>>> kvm *kvm, struct kvm_device_attr *attr) >>>>>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask)); >>>>>>>>           VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping >>>>>>>> support"); >>>>>>>>           break; >>>>>>>> +    case KVM_S390_VM_CRYPTO_INTERPRET_AP: >>>>>>>> +        if (attr->addr) { >>>>>>>> +            if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP)) >>>>>>> Unlock mutex before returning? >>>>>>> >>>>>>> Maybe flip conditions (don't allow manipulating apie if feature >>>>>>> not there). >>>>>>> Clearing the anyways clear apie if feature not there ain't too >>>>>>> bad, but >>>>>>> rejecting the operation appears nicer to me. >>>>>>> >>>>>>>> +                return -EOPNOTSUPP; >>>>>>>> +            kvm->arch.crypto.apie = 1; >>>>>>>> +            VM_EVENT(kvm, 3, "%s", >>>>>>>> +                 "ENABLE: AP interpretive execution"); >>>>>>>> +        } else { >>>>>>>> +            kvm->arch.crypto.apie = 0; >>>>>>>> +            VM_EVENT(kvm, 3, "%s", >>>>>>>> +                 "DISABLE: AP interpretive execution"); >>>>>>>> +        } >>>>>>>> +        break; >>>>>>>>       default: >>>>>>>>           mutex_unlock(&kvm->lock); >>>>>>>>           return -ENXIO; >>>>>>> I wonder how the loop after this switch works for >>>>>>> KVM_S390_VM_CRYPTO_INTERPRET_AP: >>>>>>> >>>>>>>          kvm_for_each_vcpu(i, vcpu, kvm) { >>>>>>>                  kvm_s390_vcpu_crypto_setup(vcpu); >>>>>>>                  exit_sie(vcpu); >>>>>>>          } >>>>>>> >>>>>>>  From not doing something like for KVM_S390_VM_CRYPTO_INTERPRET_AP >>>>>>> >>>>>>>          if (kvm->created_vcpus) { >>>>>>>                  mutex_unlock(&kvm->lock); >>>>>>>                  return -EBUSY; >>>>>>> and from the aforementioned loop I guess ECA.28 can be changed >>>>>>> for a running guest. >>>>>>> >>>>>>> If there are running vcpus when KVM_S390_VM_CRYPTO_INTERPRET_AP is >>>>>>> changed (set) these will be taken out of SIE by exit_sie(). Then >>>>>>> for the >>>>>>> corresponding threads the control probably goes to QEMU (the >>>>>>> emulator in >>>>>>> the userspace). And it puts that vcpu back into the SIE, and >>>>>>> then that >>>>>>> cpu starts acting according to the new ECA.28 value. While other >>>>>>> vcpus >>>>>>> may still work with the old value of ECA.28. >>>>>>> >>>>>>> I'm not saying what I describe above is necessarily something >>>>>>> broken. >>>>>>> But I would like to have it explained, why is it OK -- provided >>>>>>> I did not >>>>>>> make any errors in my reasoning (assumptions included). >>>>>>> >>>>>>> Can you help me understand this code? >>>>>>> >>>>>>> Regards, >>>>>>> Halil >>>>>>> >>>>>>> [..] >>>>>>> >>>>>> >>>>>> I have the same concerns as Halil. >>>>>> >>>>>> We do not need to change the virtulization type >>>>>> (hardware/software) on the fly for the current use case. >>>>>> >>>>>> Couldn't we delay this until we have one and in between only make >>>>>> the vCPU hotplug clean? >>>>>> >>>>>> We only need to let the door open for the day we have such a use >>>>>> case. >>>>> Are you suggesting this code be removed? If so, then where and >>>>> under what conditions would >>>>> you suggest setting ECA.28 given you objected to setting it based >>>>> on whether the >>>>> AP feature is installed? >>>> >>>> I would only call kvm_s390_vcpu_crypto_setup() from inside >>>> kvm_arch_vcpu_init() >>>> as it is already. >>> It is not called from kvm_arch_vcpu_init(), it is called from >>> kvm_arch_vcpu_setup(). >> >> hum, sorry for this. >> However, the idea pertains, not to call this function from inside an >> ioctl changing crypto parameters, but only during vcpu creation. > Unfortunately, the ioctl does not get called until after the vcpus are > created (see my comments below) That is why I think you should not change the ECA field from the crypto ioctl but only during the vcpu initialization phase. >> >> >> >>> Also, >>> this loop was already here, I did not put it in. Assuming whomever >>> put it there did so >>> for a reason, it is not my place to remove it. According to a trace >>> I ran, the calls to this >>> function occur after the vcpus are created. Consequently, the >>> kvm_s390_vcpu_crypto_setup() >>> function would not be called without the loop and neither the key >>> wrapping support nor the >>> ECA_APIE would be configured in the vcpu's SIE descriptor. >>> >>> If you have a better idea for where/how to set this flag, I'm all >>> ears. It would be nice if it could be set before the vcpus are >>> created, but I haven't >>> found a good candidate. I suspect that the loop was put in to make >>> sure that all vcpus >>> get updated regardless of whether they are running or not, but I >>> don't know what happens >>> after a vcpu is kicked out of SIE. I suspect, as Halil surmised, >>> that QEMU >>> restores the vcpus to SIE. This would seemingly cause the >>> kvm_arch_vcpu_setup() to get >>> called at which time the ECA_APIE value as well as the key wrapping >>> values will get set. >>> If somebody has knowledge of the flow here, please feel free to >>> pitch in. >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> Pierre >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -- Pierre Morel Linux/KVM/QEMU in Böblingen - Germany