Received: by 10.213.65.68 with SMTP id h4csp89542imn; Thu, 15 Mar 2018 10:24:01 -0700 (PDT) X-Google-Smtp-Source: AG47ELsBXY50fi+pGFBZt46LmvfuGrbPmt59u6Jp1/kdluTsAlNua4l9k6Es1oxm9LtIeEtRDIPP X-Received: by 10.101.88.76 with SMTP id s12mr5914211pgr.423.1521134641522; Thu, 15 Mar 2018 10:24:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1521134641; cv=none; d=google.com; s=arc-20160816; b=mXzzFIzQZwLHtHJkGSdL2wBXcTHXXzgQj1B2ABDL/EQP4izqXcCyFVccVcRqnCiSDp g+2u9XbGZCIsjNKJvHu7/CZNtNXgbx6yLd8Eatxpr0ToMvk4H4kyxHClNsDZ5aAOGGtK 23jHsFeJYQgKD8BG7i8rV3fyebXZ9Zlij4EKZ6ylV8bl8FwhTg4t5pLAH6FUWbeqY89j 0k1jOScCWaWzHlw3134pYUvyJXQGSwO/1/YWKexLzn5hNZhKrNkcaRM24USgbxHwMBRn +FpOubzVPGbcPFPd/hs1VRelQG27lYcRMr+FK/Q467PDxzvkKSRk2DEHzD7DJuKqto2w B2oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:content-language :content-transfer-encoding:in-reply-to:mime-version:user-agent:date :from:references:cc:to:subject:arc-authentication-results; bh=57NkNxOLseaNvI2inXo6u6eeS7oWvi9qena3LNZB7aU=; b=BQcqMFo0bitBMxJARYWSCG2c8WRzzcyOFImrqQqVQCtdPbHRDqzNGzfsbFJsxy9Je8 yhexagRuhF02s1XNh17uy1k6FJiNS4e6K0eKdFc5bBFFhfqpTmDt5tu6BOgdsXPALPDz G/p5awGnFnCDzXv7u/UAPGzGqQVVjPcaKiqH+0foUzdB8w+3DTqYyFiRGVyF5HPLHo6S shwtoqVJngTXSeRnHtEvUcI3hOQqilXB068+QFyQj8rRtnt3ghnzukmo3tjwsccYQ38h DH0cIBx2vKUEoAb7XKiAGoyviaNMQX5uMXlQXfW83Ly3ZIofzFE5NW7NNk7EfcoTIXwg 5g1Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q24-v6si4266542pls.600.2018.03.15.10.23.46; Thu, 15 Mar 2018 10:24:01 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752232AbeCORWG (ORCPT + 99 others); Thu, 15 Mar 2018 13:22:06 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:42362 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752130AbeCORWE (ORCPT ); Thu, 15 Mar 2018 13:22:04 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w2FHLAkj127648 for ; Thu, 15 Mar 2018 13:22:04 -0400 Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.149]) by mx0a-001b2d01.pphosted.com with ESMTP id 2gqvg81x4k-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Thu, 15 Mar 2018 13:22:03 -0400 Received: from localhost by e31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 15 Mar 2018 11:22:03 -0600 Received: from b03cxnp08028.gho.boulder.ibm.com (9.17.130.20) by e31.co.us.ibm.com (192.168.1.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 15 Mar 2018 11:21:59 -0600 Received: from b03ledav002.gho.boulder.ibm.com (b03ledav002.gho.boulder.ibm.com [9.17.130.233]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w2FHLwUJ12386708; Thu, 15 Mar 2018 10:21:58 -0700 Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0F25A136044; Thu, 15 Mar 2018 11:21:58 -0600 (MDT) Received: from oc8043147753.ibm.com (unknown [9.80.217.151]) by b03ledav002.gho.boulder.ibm.com (Postfix) with ESMTP id 67B1513603C; Thu, 15 Mar 2018 11:21:55 -0600 (MDT) Subject: Re: [PATCH v3 04/14] KVM: s390: device attribute to set AP interpretive execution To: Pierre Morel , Halil Pasic , linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@de.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, kwankhede@nvidia.com, bjsdjshi@linux.vnet.ibm.com, pbonzini@redhat.com, alex.williamson@redhat.com, alifm@linux.vnet.ibm.com, mjrosato@linux.vnet.ibm.com, jjherne@linux.vnet.ibm.com, thuth@redhat.com, berrange@redhat.com, fiuczy@linux.vnet.ibm.com, buendgen@de.ibm.com References: <1521051954-25715-1-git-send-email-akrowiak@linux.vnet.ibm.com> <1521051954-25715-5-git-send-email-akrowiak@linux.vnet.ibm.com> <21bd029b-3500-3461-ce98-68ad3ae9b647@linux.vnet.ibm.com> <46a7e838-2be2-9587-6eb2-3bba95485609@linux.vnet.ibm.com> <5ed8017b-0168-9a50-234b-cfe9258eab72@linux.vnet.ibm.com> From: Tony Krowiak Date: Thu, 15 Mar 2018 13:21:54 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <5ed8017b-0168-9a50-234b-cfe9258eab72@linux.vnet.ibm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 18031517-8235-0000-0000-00000D2A8781 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008679; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000254; SDB=6.01003484; UDB=6.00510698; IPR=6.00782810; MB=3.00020054; MTD=3.00000008; XFM=3.00000015; UTC=2018-03-15 17:22:02 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18031517-8236-0000-0000-000040130098 Message-Id: <17683324-f6e4-4328-54c1-1fce572faecd@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-03-15_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1803150189 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/15/2018 11:45 AM, Pierre Morel wrote: > On 15/03/2018 16:26, Tony Krowiak wrote: >> On 03/15/2018 09:00 AM, Pierre Morel wrote: >>> On 14/03/2018 22:57, Halil Pasic wrote: >>>> >>>> On 03/14/2018 07:25 PM, Tony Krowiak wrote: >>>>> The VFIO AP device model exploits interpretive execution of AP >>>>> instructions (APIE) to provide guests passthrough access to AP >>>>> devices. This patch introduces a new device attribute in the >>>>> KVM_S390_VM_CRYPTO device attribute group to set APIE from >>>>> the VFIO AP device defined on the guest. >>>>> >>>>> Signed-off-by: Tony Krowiak >>>>> --- >>>> [..] >>>> >>>>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c >>>>> index a60c45b..bc46b67 100644 >>>>> --- a/arch/s390/kvm/kvm-s390.c >>>>> +++ b/arch/s390/kvm/kvm-s390.c >>>>> @@ -815,6 +815,19 @@ static int kvm_s390_vm_set_crypto(struct kvm >>>>> *kvm, struct kvm_device_attr *attr) >>>>> sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask)); >>>>> VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support"); >>>>> break; >>>>> + case KVM_S390_VM_CRYPTO_INTERPRET_AP: >>>>> + if (attr->addr) { >>>>> + if (!test_kvm_cpu_feat(kvm, KVM_S390_VM_CPU_FEAT_AP)) >>>> Unlock mutex before returning? >>>> >>>> Maybe flip conditions (don't allow manipulating apie if feature not >>>> there). >>>> Clearing the anyways clear apie if feature not there ain't too bad, >>>> but >>>> rejecting the operation appears nicer to me. >>>> >>>>> + return -EOPNOTSUPP; >>>>> + kvm->arch.crypto.apie = 1; >>>>> + VM_EVENT(kvm, 3, "%s", >>>>> + "ENABLE: AP interpretive execution"); >>>>> + } else { >>>>> + kvm->arch.crypto.apie = 0; >>>>> + VM_EVENT(kvm, 3, "%s", >>>>> + "DISABLE: AP interpretive execution"); >>>>> + } >>>>> + break; >>>>> default: >>>>> mutex_unlock(&kvm->lock); >>>>> return -ENXIO; >>>> I wonder how the loop after this switch works for >>>> KVM_S390_VM_CRYPTO_INTERPRET_AP: >>>> >>>> kvm_for_each_vcpu(i, vcpu, kvm) { >>>> kvm_s390_vcpu_crypto_setup(vcpu); >>>> exit_sie(vcpu); >>>> } >>>> >>>> From not doing something like for KVM_S390_VM_CRYPTO_INTERPRET_AP >>>> >>>> if (kvm->created_vcpus) { >>>> mutex_unlock(&kvm->lock); >>>> return -EBUSY; >>>> and from the aforementioned loop I guess ECA.28 can be changed >>>> for a running guest. >>>> >>>> If there are running vcpus when KVM_S390_VM_CRYPTO_INTERPRET_AP is >>>> changed (set) these will be taken out of SIE by exit_sie(). Then >>>> for the >>>> corresponding threads the control probably goes to QEMU (the >>>> emulator in >>>> the userspace). And it puts that vcpu back into the SIE, and then that >>>> cpu starts acting according to the new ECA.28 value. While other >>>> vcpus >>>> may still work with the old value of ECA.28. >>>> >>>> I'm not saying what I describe above is necessarily something broken. >>>> But I would like to have it explained, why is it OK -- provided I >>>> did not >>>> make any errors in my reasoning (assumptions included). >>>> >>>> Can you help me understand this code? >>>> >>>> Regards, >>>> Halil >>>> >>>> [..] >>>> >>> >>> I have the same concerns as Halil. >>> >>> We do not need to change the virtulization type >>> (hardware/software) on the fly for the current use case. >>> >>> Couldn't we delay this until we have one and in between only make >>> the vCPU hotplug clean? >>> >>> We only need to let the door open for the day we have such a use case. >> Are you suggesting this code be removed? If so, then where and under >> what conditions would >> you suggest setting ECA.28 given you objected to setting it based on >> whether the >> AP feature is installed? > > I would only call kvm_s390_vcpu_crypto_setup() from inside > kvm_arch_vcpu_init() > as it is already. It is not called from kvm_arch_vcpu_init(), it is called from kvm_arch_vcpu_setup(). Also, this loop was already here, I did not put it in. Assuming whomever put it there did so for a reason, it is not my place to remove it. According to a trace I ran, the calls to this function occur after the vcpus are created. Consequently, the kvm_s390_vcpu_crypto_setup() function would not be called without the loop and neither the key wrapping support nor the ECA_APIE would be configured in the vcpu's SIE descriptor. If you have a better idea for where/how to set this flag, I'm all ears. It would be nice if it could be set before the vcpus are created, but I haven't found a good candidate. I suspect that the loop was put in to make sure that all vcpus get updated regardless of whether they are running or not, but I don't know what happens after a vcpu is kicked out of SIE. I suspect, as Halil surmised, that QEMU restores the vcpus to SIE. This would seemingly cause the kvm_arch_vcpu_setup() to get called at which time the ECA_APIE value as well as the key wrapping values will get set. If somebody has knowledge of the flow here, please feel free to pitch in. > > > >>> >>> >>> Pierre >>> >>> >>> >> >