Received: by 10.192.165.156 with SMTP id m28csp258540imm; Tue, 17 Apr 2018 09:39:28 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/HJokU8wGnJerWWw2tmE4I6oaUsFSMHB/mNmx/8qWyb10f/AsA76imUwb6QQHmv5oM09A2 X-Received: by 10.101.99.22 with SMTP id g22mr2401767pgv.410.1523983168309; Tue, 17 Apr 2018 09:39:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523983168; cv=none; d=google.com; s=arc-20160816; b=ibeU0nZEiZIllhUGn3WGH3IdfFFTmXmX/x+y+HxViPZOoFKlb/jUXwkwpIA556CQzc GJq6K1OSZIltYw3WADE7vJ5UWCSTYtK5qTegAZK/AIU0j5jkqTNs42zoATZE/JoMzoBp fjtOn3hT5BBgNndthkPP8bp+s1aCDHNQEu9NbwQH7tuvXe6nxCdlfy+2Paz3B7zEQIfj BBuFkSfztPg7+Iro8j20OjNoPBeOvhwIHhr1xnO0mdHepXhweuQMvhdAClGQjrDjsNjC 6ZdEi35GXydeQG0k56rzsMGEae+mKDYlW0HufAv5gwPGdDZA3B612xeOFIr6XUaMUSLH vaGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:content-language :content-transfer-encoding:in-reply-to:mime-version:user-agent:date :from:references:cc:to:subject:arc-authentication-results; bh=nUfwYZzUrJU9rTSmw8iZ5SGvwfziYyXodHcC2h79dMg=; b=U7AFMcdWZRP7AL/opj1cFwv/jF0nY669/QaIHSr2SeCCMcSCGxrAJuDV2BEeqc1UgZ s5uIs1iARi8sIrc06O6b/wLmKmPNmcsJZ6+DE+GIPkVQP92rMRXqU1MXnyNTFTJLP8Iy O+ElVx3H8syFZOpXC3vwpZXgm3tGbGQrbUOp97Ihh4+pINPm42oB16o/isvoaVBLfbCM 5C/JEgoP5qrIuVIkiTYUDmQTQrYcgg6oSTULn59GQVHK/flB7tCeQRO+c3YLS/V6XuDD HTvsSaYHGKEOXPK5sfGSCH5xAvzyXopAxziUTyRE3fU2h0t5bufPXKcQ9RE34ZxGx/kJ 44IA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o128si12431191pfg.5.2018.04.17.09.39.13; Tue, 17 Apr 2018 09:39:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755231AbeDQQgq (ORCPT + 99 others); Tue, 17 Apr 2018 12:36:46 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:59336 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754843AbeDQQgm (ORCPT ); Tue, 17 Apr 2018 12:36:42 -0400 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w3HGZRMT008522 for ; Tue, 17 Apr 2018 12:36:42 -0400 Received: from e14.ny.us.ibm.com (e14.ny.us.ibm.com [129.33.205.204]) by mx0a-001b2d01.pphosted.com with ESMTP id 2hdjd18128-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Tue, 17 Apr 2018 12:36:41 -0400 Received: from localhost by e14.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 17 Apr 2018 12:36:40 -0400 Received: from b01cxnp22036.gho.pok.ibm.com (9.57.198.26) by e14.ny.us.ibm.com (146.89.104.201) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 17 Apr 2018 12:36:35 -0400 Received: from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com [9.57.199.107]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w3HGaYY957278500; Tue, 17 Apr 2018 16:36:34 GMT Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E9625124037; Tue, 17 Apr 2018 13:38:36 -0400 (EDT) Received: from oc8043147753.ibm.com (unknown [9.85.136.174]) by b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP id C5FD5124035; Tue, 17 Apr 2018 13:38:35 -0400 (EDT) Subject: Re: [PATCH v4 13/15] KVM: s390: configure the guest's AP devices To: Pierre Morel , linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@de.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, kwankhede@nvidia.com, bjsdjshi@linux.vnet.ibm.com, pbonzini@redhat.com, alex.williamson@redhat.com, alifm@linux.vnet.ibm.com, mjrosato@linux.vnet.ibm.com, jjherne@linux.vnet.ibm.com, thuth@redhat.com, pasic@linux.vnet.ibm.com, berrange@redhat.com, fiuczy@linux.vnet.ibm.com, buendgen@de.ibm.com References: <1523827345-11600-1-git-send-email-akrowiak@linux.vnet.ibm.com> <1523827345-11600-14-git-send-email-akrowiak@linux.vnet.ibm.com> <2e7387b9-ceb4-cdca-56ac-f5344084e10a@linux.vnet.ibm.com> <105bce81-9359-6096-88ce-01ec1aef523e@linux.vnet.ibm.com> From: Tony Krowiak Date: Tue, 17 Apr 2018 12:36:33 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <105bce81-9359-6096-88ce-01ec1aef523e@linux.vnet.ibm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 18041716-0052-0000-0000-000002DD23B2 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008871; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000257; SDB=6.01019289; UDB=6.00519990; IPR=6.00798531; MB=3.00020619; MTD=3.00000008; XFM=3.00000015; UTC=2018-04-17 16:36:38 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18041716-0053-0000-0000-00005C5D99A6 Message-Id: <6eb66344-d200-81d8-6533-aa1f9b53271e@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-04-17_08:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1804170145 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/17/2018 12:18 PM, Pierre Morel wrote: > On 17/04/2018 18:08, Tony Krowiak wrote: >> On 04/16/2018 09:05 AM, Pierre Morel wrote: >>> On 15/04/2018 23:22, Tony Krowiak wrote: >>>> Registers a group notifier during the open of the mediated >>>> matrix device to get information on KVM presence through the >>>> VFIO_GROUP_NOTIFY_SET_KVM event. When notified, the pointer >>>> to the kvm structure is saved inside the mediated matrix >>>> device. Once the VFIO AP device driver has access to KVM, >>>> access to the APs can be configured for the guest. >>>> >>>> Access to APs is configured when the file descriptor for the >>>> mediated matrix device is opened by userspace. The items to be >>>> configured are: >>>> >>>> 1. The ECA.28 bit in the SIE state description determines whether >>>> AP instructions are interpreted by the hardware or intercepted. >>>> The VFIO AP device driver relies interpretive execution of >>>> AP instructions so the ECA.28 bit will be set >>>> >>>> 2. Guest access to AP adapters, usage domains and control domains >>>> is controlled by three bit masks referenced from the >>>> Crypto Control Block (CRYCB) referenced from the guest's SIE state >>>> description: >>>> >>>> * The AP Mask (APM) controls access to the AP adapters. Each bit >>>> in the APM represents an adapter number - from most significant >>>> to least significant bit - from 0 to 255. The bits in the APM >>>> are set according to the adapter numbers assigned to the >>>> mediated >>>> matrix device via its 'assign_adapter' sysfs attribute file. >>>> >>>> * The AP Queue (AQM) controls access to the AP queues. Each bit >>>> in the AQM represents an AP queue index - from most significant >>>> to least significant bit - from 0 to 255. A queue index >>>> references >>>> a specific domain and is synonymous with the domian number. The >>>> bits in the AQM are set according to the domain numbers assigned >>>> to the mediated matrix device via its 'assign_domain' sysfs >>>> attribute file. >>>> >>>> * The AP Domain Mask (ADM) controls access to the AP control >>>> domains. >>>> Each bit in the ADM represents a control domain - from most >>>> significant to least significant bit - from 0-255. The >>>> bits in the ADM are set according to the domain numbers assigned >>>> to the mediated matrix device via its 'assign_control_domain' >>>> sysfs attribute file. >>>> >>>> Signed-off-by: Tony Krowiak >>>> --- >>>> drivers/s390/crypto/vfio_ap_ops.c | 50 >>>> +++++++++++++++++++++++++++++++++ >>>> drivers/s390/crypto/vfio_ap_private.h | 2 + >>>> 2 files changed, 52 insertions(+), 0 deletions(-) >>>> >>>> diff --git a/drivers/s390/crypto/vfio_ap_ops.c >>>> b/drivers/s390/crypto/vfio_ap_ops.c >>>> index bc2b05e..e3ff5ab 100644 >>>> --- a/drivers/s390/crypto/vfio_ap_ops.c >>>> +++ b/drivers/s390/crypto/vfio_ap_ops.c >>>> @@ -53,6 +53,54 @@ static int vfio_ap_mdev_remove(struct >>>> mdev_device *mdev) >>>> return 0; >>>> } >>>> >>>> +static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, >>>> + unsigned long action, void *data) >>>> +{ >>>> + struct ap_matrix_mdev *matrix_mdev; >>>> + >>>> + if (action == VFIO_GROUP_NOTIFY_SET_KVM) { >>>> + matrix_mdev = container_of(nb, struct ap_matrix_mdev, >>>> + group_notifier); >>>> + matrix_mdev->kvm = data; >>>> + } >>>> + >>>> + return NOTIFY_OK; >>>> +} >>>> + >>>> +static int vfio_ap_mdev_open(struct mdev_device *mdev) >>>> +{ >>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); >>>> + unsigned long events; >>>> + int ret; >>>> + >>>> + matrix_mdev->group_notifier.notifier_call = >>>> vfio_ap_mdev_group_notifier; >>>> + events = VFIO_GROUP_NOTIFY_SET_KVM; >>>> + >>>> + ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY, >>>> + &events, &matrix_mdev->group_notifier); >>>> + if (ret) >>>> + return ret; >>>> + >>>> + ret = kvm_ap_interpret_instructions(matrix_mdev->kvm, true); >>> >>> Do you need this call ? >>> apie is always enabled in KVM if AP instructions are available. >> >> I suppose we don't, in which case we don't need the >> kvm_ap_interpret_instructions() >> function either ... at least not until we implement interception. >> >>> >>> >>> Setting or not the interpretation is done for the VM in a all. >>> It is not the right place to do it here since open is device dependent. >> >> As I stated above, at this time we probably do not need this, however; >> that will not always be the case. The setting is and always will be >> for the >> VM in all - unless the architecture changes - because it is >> controlled by a >> single bit (ECA.28). If you recall, I originally set interpretation >> in the >> vfio_ap device driver when notified of the VFIO_GROUP_NOTIFY_SET_KVM >> event. >> I believe ultimately that it is the device driver that should set the >> value >> for apie. >> >> >> >>> >>> >>> Or we only have one device in the VM at a time. >>> In this case, shouldn't we make it official by returning -EEXIST for >>> the second call? >> >> We do allow only one vfio-ap device at a time. QEMU will allow only >> one vfio-ap device >> to be configured for a guest. Should we also put a check in here? > > QEMU is not the only possible user of this interface. True .... I will put a check in here to make sure only one device is created. > > >> >>> >>> >>> >>>> + if (ret) >>>> + return ret; >>>> + >>>> + ret = kvm_ap_configure_matrix(matrix_mdev->kvm, >>>> + matrix_mdev->matrix); >>>> + >>>> + return ret; >>>> +} >>>> + >>>> +static void vfio_ap_mdev_release(struct mdev_device *mdev) >>>> +{ >>>> + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); >>>> + >>>> + kvm_ap_deconfigure_matrix(matrix_mdev->kvm); >>>> + kvm_ap_interpret_instructions(matrix_mdev->kvm, false); >>> >>> This call clears the apie in KVM. >>> This is only OK if we have a single device present until the end of >>> the VM, >>> otherwise AP instructions in the guest will fail after the release >>> until the end of the VM >>> or until a new device is plugged. >> >> See Message ID: >> <1523819244-29954-5-git-send-email-akrowiak@linux.vnet.ibm.com> on the >> qemu mailing list. There will be only one vfio-ap device allowed for >> the MVP model. > > dito. > Anyone can write a userland application using this interface. See comments above, not to mention I will probably remove this call. > > <2b053349-071e-17ed-6ebd-a37bcfd2f330@linux.vnet.ibm.com> >> >>> >>> >>> >>>> + vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY, >>>> + &matrix_mdev->group_notifier); >>>> +} >>>> + >>>> static ssize_t name_show(struct kobject *kobj, struct device >>>> *dev, char *buf) >>>> { >>>> return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT); >>>> @@ -754,6 +802,8 @@ static ssize_t matrix_show(struct device *dev, >>>> struct device_attribute *attr, >>>> .mdev_attr_groups = vfio_ap_mdev_attr_groups, >>>> .create = vfio_ap_mdev_create, >>>> .remove = vfio_ap_mdev_remove, >>>> + .open = vfio_ap_mdev_open, >>>> + .release = vfio_ap_mdev_release, >>>> }; >>>> >>>> int vfio_ap_mdev_register(struct ap_matrix *ap_matrix) >>>> diff --git a/drivers/s390/crypto/vfio_ap_private.h >>>> b/drivers/s390/crypto/vfio_ap_private.h >>>> index f248faf..48e2806 100644 >>>> --- a/drivers/s390/crypto/vfio_ap_private.h >>>> +++ b/drivers/s390/crypto/vfio_ap_private.h >>>> @@ -31,6 +31,8 @@ struct ap_matrix { >>>> >>>> struct ap_matrix_mdev { >>>> struct kvm_ap_matrix *matrix; >>>> + struct notifier_block group_notifier; >>>> + struct kvm *kvm; >>>> }; >>>> >>>> static inline struct ap_matrix *to_ap_matrix(struct device *dev) >>> >>> >> >