Received: by 10.223.185.116 with SMTP id b49csp6398027wrg; Wed, 28 Feb 2018 08:45:11 -0800 (PST) X-Google-Smtp-Source: AH8x226jV0Uap163CtIRAro/cwWbDlikZRgDLTrAUg2Wwe5WUoD0vMWuZV4j3CaXFHLU8C+Aoceu X-Received: by 10.99.153.1 with SMTP id d1mr14531540pge.338.1519836311482; Wed, 28 Feb 2018 08:45:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1519836311; cv=none; d=google.com; s=arc-20160816; b=lcxhP+srPCQGOU7MnnLUwH/emytrRtdeVPgtqTBNr/KGNYFxDeaCYsb4vEOFpfy7Hh 4KVp7IkcAQyVTKUfgsTBGIgDY0W9EgpOUXKYi7YAp3ZzspjmaqS9uRhB3b0zEWdf+scF xFIwYPVMJkKYCubXQxBlWArn2zT8+DvbOz1OTJES73H1rYDQw1cxOJRMDOoCq0tLJmJa Z6Ec9Ke3RJJceLHDdwpq5svYmMx9c06Y69q89ZSsrfm1csSWW+Xih7M6OxvNrf3p4xEE I4Fk5EtVYnC9qLajl4Axh+eqtJVQJWyIgnUMVYGhfOkZsQMOGg3fJ08rukef+FSTMYwf rRRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:content-language :content-transfer-encoding:in-reply-to:mime-version:user-agent:date :from:references:cc:to:subject:arc-authentication-results; bh=5HwnPE867UPA1CDxS8AG4Cz77aB0Fo9bSiLlVFn7Bbc=; b=SXib1nSNJeUxzwaTkDGPR6wdk7RMzoMEuTSqKggROj9md2625CMBZEjXU6fwPoGGCc I7vZbkojo6rdWxhEFQD5Spz6T+DStFTePhz7fQMyfDmDISWtBZhEQG5CuYNMX1VZ3ibh PsYKrln4VIKAK3hBcrwa/OlvJUhTmx+2ZJu66dAfJWdWSibTV4TI7VFhIVlTdESj8Xii 8cI46SBY9MMZD298STWnotv9WOYfITANnX+8w+jYR39zkczEZ1XGGy91vyREceUs/Flx dTbqkyzGkMGukrU962sFHuRvvV9C8pGLgBv9mr/z9VwU9IgzDeYUZeW2FnX6RyHZ6e1P Tyuw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m12-v6si1523601pln.400.2018.02.28.08.44.56; Wed, 28 Feb 2018 08:45:11 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753223AbeB1Qnu (ORCPT + 99 others); Wed, 28 Feb 2018 11:43:50 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:33760 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752679AbeB1Qnp (ORCPT ); Wed, 28 Feb 2018 11:43:45 -0500 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w1SGe9Vs128520 for ; Wed, 28 Feb 2018 11:43:45 -0500 Received: from e18.ny.us.ibm.com (e18.ny.us.ibm.com [129.33.205.208]) by mx0b-001b2d01.pphosted.com with ESMTP id 2gdwyyf46u-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 28 Feb 2018 11:43:44 -0500 Received: from localhost by e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 28 Feb 2018 11:43:43 -0500 Received: from b01cxnp22035.gho.pok.ibm.com (9.57.198.25) by e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 28 Feb 2018 11:43:39 -0500 Received: from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com [9.57.199.110]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w1SGhcpw50200652; Wed, 28 Feb 2018 16:43:38 GMT Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D101FAE03C; Wed, 28 Feb 2018 11:45:01 -0500 (EST) Received: from oc8043147753.ibm.com (unknown [9.60.75.238]) by b01ledav005.gho.pok.ibm.com (Postfix) with ESMTP id AAABEAE051; Wed, 28 Feb 2018 11:45:00 -0500 (EST) Subject: Re: [PATCH v2 05/15] s390: vfio-ap: base implementation of VFIO AP device driver To: Pierre Morel , linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@de.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, kwankhede@nvidia.com, bjsdjshi@linux.vnet.ibm.com, pbonzini@redhat.com, alex.williamson@redhat.com, alifm@linux.vnet.ibm.com, mjrosato@linux.vnet.ibm.com, jjherne@linux.vnet.ibm.com, thuth@redhat.com, pasic@linux.vnet.ibm.com, fiuczy@linux.vnet.ibm.com, buendgen@de.ibm.com References: <1519741693-17440-1-git-send-email-akrowiak@linux.vnet.ibm.com> <1519741693-17440-6-git-send-email-akrowiak@linux.vnet.ibm.com> From: Tony Krowiak Date: Wed, 28 Feb 2018 11:43:37 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 18022816-0044-0000-0000-000003EC10D6 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008602; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000254; SDB=6.00996385; UDB=6.00506542; IPR=6.00775707; MB=3.00019780; MTD=3.00000008; XFM=3.00000015; UTC=2018-02-28 16:43:41 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18022816-0045-0000-0000-0000081C10D9 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-28_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802280202 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/28/2018 10:33 AM, Pierre Morel wrote: > On 27/02/2018 15:28, Tony Krowiak wrote: >> Introduces a new AP device driver. This device driver >> is built on the VFIO mediated device framework. The framework >> provides sysfs interfaces that facilitate passthrough >> access by guests to devices installed on the linux host. >> >> The VFIO AP device driver will serve two purposes: >> >> 1. Provide the interfaces to reserve AP devices for exclusive >> use by KVM guests. This is accomplished by unbinding the >> devices to be reserved for guest usage from the default AP >> device driver and binding them to the VFIO AP device driver. >> >> 2. Implements the functions, callbacks and sysfs attribute >> interfaces required to create one or more VFIO mediated >> devices each of which will be used to configure the AP >> matrix for a guest and serve as a file descriptor >> for facilitating communication between QEMU and the >> VFIO AP device driver. >> >> When the VFIO AP device driver is initialized: >> >> * It registers with the AP bus for control of type 10 (CEX4 >> and newer) AP queue devices. The probe and remove callbacks >> will be provided to support the binding/unbinding of >> AP queue devices to/from the VFIO AP device driver. >> >> * Creates a /sys/devices/vfio-ap/matrix device to hold >> the APQNs of the AP devices bound to the VFIO >> AP device driver and serves as the parent of the >> mediated devices created for each guest. >> >> Signed-off-by: Tony Krowiak >> --- >> MAINTAINERS | 2 + >> arch/s390/Kconfig | 8 ++ >> arch/s390/configs/default_defconfig | 3 + >> arch/s390/configs/gcov_defconfig | 3 + >> arch/s390/configs/performance_defconfig | 3 + >> arch/s390/defconfig | 3 + >> drivers/s390/crypto/Makefile | 4 + >> drivers/s390/crypto/vfio_ap_drv.c | 134 >> +++++++++++++++++++++++++++++++ >> drivers/s390/crypto/vfio_ap_private.h | 20 +++++ >> include/uapi/linux/vfio.h | 2 + >> 10 files changed, 182 insertions(+), 0 deletions(-) >> create mode 100644 drivers/s390/crypto/vfio_ap_drv.c >> create mode 100644 drivers/s390/crypto/vfio_ap_private.h >> >> diff --git a/MAINTAINERS b/MAINTAINERS >> index 4acf7c2..a2f232d 100644 >> --- a/MAINTAINERS >> +++ b/MAINTAINERS >> @@ -11884,6 +11884,8 @@ W: >> http://www.ibm.com/developerworks/linux/linux390/ >> S: Supported >> F: arch/s390/include/asm/kvm/kvm-ap.h >> F: arch/s390/kvm/kvm-ap.c >> +F: drivers/s390/crypto/vfio_ap_drv.c >> +F: drivers/s390/crypto/vfio_ap_private.h >> >> S390 ZFCP DRIVER >> M: Steffen Maier >> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig >> index cbe1d97..9f23caf 100644 >> --- a/arch/s390/Kconfig >> +++ b/arch/s390/Kconfig >> @@ -771,6 +771,14 @@ config VFIO_CCW >> To compile this driver as a module, choose M here: the >> module will be called vfio_ccw. >> >> +config VFIO_AP >> + def_tristate m >> + prompt "Support for virtual Adjunct Processor device interface" > > The VFIO AP devices are not virtual. > What about > "VFIO support for AP devices" Sounds good. > >> + depends on ZCRYPT && VFIO_MDEV_DEVICE >> + help >> + driver grants access to Adjunct Processor (AP) devices >> + via the VFIO mediated device interface. >> + >> endmenu >> >> menu "Dump support" >> diff --git a/arch/s390/configs/default_defconfig >> b/arch/s390/configs/default_defconfig >> index 5af8458..40fa3f6 100644 >> --- a/arch/s390/configs/default_defconfig >> +++ b/arch/s390/configs/default_defconfig > > Not sure that this file belongs to this patch Neither am I, but at the time I inserted this here - well before August of last year - I was using vfio-ccw as a model. If someone can verify this does not belong here, I'd be more than happy to remove it. > > >> @@ -719,3 +719,6 @@ CONFIG_APPLDATA_BASE=y >> CONFIG_KVM=m >> CONFIG_KVM_S390_UCONTROL=y >> CONFIG_VHOST_NET=m >> +VFIO_MDEV=m >> +VFIO_MDEV_DEVICE=m >> +CONFIG_VFIO_AP=m > > What is your goal when modifying this three files? > Could you add a comment in the commit message? As stated above, this was originally based on the vfio-ccw model and has been in the patch series since its inception. I'd be happy to remove it if it is not necessary. > > > >> diff --git a/arch/s390/configs/gcov_defconfig >> b/arch/s390/configs/gcov_defconfig >> index d52eafe..377d40f 100644 >> --- a/arch/s390/configs/gcov_defconfig >> +++ b/arch/s390/configs/gcov_defconfig >> @@ -659,3 +659,6 @@ CONFIG_APPLDATA_BASE=y >> CONFIG_KVM=m >> CONFIG_KVM_S390_UCONTROL=y >> CONFIG_VHOST_NET=m >> +VFIO_MDEV=m >> +VFIO_MDEV_DEVICE=m >> +CONFIG_VFIO_AP=m >> diff --git a/arch/s390/configs/performance_defconfig >> b/arch/s390/configs/performance_defconfig >> index 20ed149..f0c52df 100644 >> --- a/arch/s390/configs/performance_defconfig >> +++ b/arch/s390/configs/performance_defconfig >> @@ -657,3 +657,6 @@ CONFIG_APPLDATA_BASE=y >> CONFIG_KVM=m >> CONFIG_KVM_S390_UCONTROL=y >> CONFIG_VHOST_NET=m >> +VFIO_MDEV=m >> +VFIO_MDEV_DEVICE=m >> +CONFIG_VFIO_AP=m >> diff --git a/arch/s390/defconfig b/arch/s390/defconfig >> index 46a3178..0996eb7 100644 >> --- a/arch/s390/defconfig >> +++ b/arch/s390/defconfig >> @@ -239,3 +239,6 @@ CONFIG_CRC7=m >> # CONFIG_XZ_DEC_ARMTHUMB is not set >> # CONFIG_XZ_DEC_SPARC is not set >> CONFIG_CMM=m >> +VFIO_MDEV=m >> +VFIO_MDEV_DEVICE=m >> +CONFIG_VFIO_AP=m >> diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile >> index b59af54..48e466e 100644 >> --- a/drivers/s390/crypto/Makefile >> +++ b/drivers/s390/crypto/Makefile >> @@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o >> zcrypt_cex2a.o zcrypt_cex4.o >> # pkey kernel module >> pkey-objs := pkey_api.o >> obj-$(CONFIG_PKEY) += pkey.o >> + >> +# adjunct processor matrix >> +vfio_ap-objs := vfio_ap_drv.o >> +obj-$(CONFIG_VFIO_AP) += vfio_ap.o >> diff --git a/drivers/s390/crypto/vfio_ap_drv.c >> b/drivers/s390/crypto/vfio_ap_drv.c >> new file mode 100644 >> index 0000000..8bb72af >> --- /dev/null >> +++ b/drivers/s390/crypto/vfio_ap_drv.c >> @@ -0,0 +1,134 @@ >> +/* >> + * VFIO based AP device driver >> + * >> + * Copyright IBM Corp. 2017 >> + * >> + * Author(s): Tony Krowiak >> + */ >> + >> +#include >> +#include >> +#include >> + >> +#include "ap_bus.h" > > Since you do it in a later patch, may be put this include of ap_bus.h > in ap_private.h now. Okay > > >> +#include "vfio_ap_private.h" >> + >> +#define VFIO_AP_ROOT_NAME "vfio_ap" >> +#define VFIO_AP_DEV_TYPE_NAME "ap_matrix" >> +#define VFIO_AP_DEV_NAME "matrix" >> + >> +MODULE_AUTHOR("IBM Corporation"); >> +MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2017"); >> +MODULE_LICENSE("GPL v2"); >> + >> +static struct device *vfio_ap_root_device; >> + >> +static struct ap_driver vfio_ap_drv; >> + >> +static struct ap_matrix *ap_matrix; >> + >> +static struct device_type vfio_ap_dev_type = { >> + .name = VFIO_AP_DEV_TYPE_NAME, >> +}; >> + >> +/* Only type 10 adapters (CEX4 and later) are supported >> + * by the AP matrix device driver >> + */ >> +static struct ap_device_id ap_queue_ids[] = { >> + { .dev_type = AP_DEVICE_TYPE_CEX4, >> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE }, >> + { .dev_type = AP_DEVICE_TYPE_CEX5, >> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE }, >> + { .dev_type = AP_DEVICE_TYPE_CEX6, >> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE }, >> + { /* end of sibling */ }, >> +}; >> + >> +MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids); >> + >> +static int vfio_ap_queue_dev_probe(struct ap_device *apdev) >> +{ >> + return 0; >> +} >> + >> +static void vfio_ap_matrix_dev_release(struct device *dev) >> +{ >> + struct ap_matrix *ap_matrix = dev_get_drvdata(dev); >> + >> + kfree(ap_matrix); >> +} >> + >> +static int vfio_ap_matrix_dev_create(void) >> +{ >> + int ret; >> + >> + vfio_ap_root_device = root_device_register(VFIO_AP_ROOT_NAME); >> + >> + ret = PTR_ERR_OR_ZERO(vfio_ap_root_device); > > IS_ERR() is enough, root_device_register() never return NULL. I searched the kernel code to look at other places the root_device_register() function is called to see how the return value is handled. I've seen all of the following used: if (IS_ERR()) ret = PTR_ERR() PTR_ERR() PTR_ERR_OR_ZERO() I'm not sure why this is a concern, but I'll use the first option above since PTR_ERR_OR_ZERO() also embeds the first option. >> + if (ret) >> + goto done; >> + >> + ap_matrix = kzalloc(sizeof(*ap_matrix), GFP_KERNEL); >> + if (!ap_matrix) { >> + ret = -ENOMEM; >> + goto matrix_alloc_err; >> + } >> + >> + ap_matrix->device.type = &vfio_ap_dev_type; >> + dev_set_name(&ap_matrix->device, "%s", VFIO_AP_DEV_NAME); >> + ap_matrix->device.parent = vfio_ap_root_device; >> + ap_matrix->device.release = vfio_ap_matrix_dev_release; >> + ap_matrix->device.driver = &vfio_ap_drv.driver; >> + >> + ret = device_register(&ap_matrix->device); >> + if (ret) >> + goto matrix_reg_err; >> + >> + goto done; >> + >> +matrix_reg_err: >> + put_device(&ap_matrix->device); >> + kfree(ap_matrix); >> + >> +matrix_alloc_err: >> + root_device_unregister(vfio_ap_root_device); >> + >> +done: >> + return ret; >> +} >> + >> +static void vfio_ap_matrix_dev_destroy(struct ap_matrix *ap_matrix) >> +{ >> + device_unregister(&ap_matrix->device); >> + root_device_unregister(vfio_ap_root_device); >> +} >> + >> +int __init vfio_ap_init(void) >> +{ >> + int ret; >> + >> + ret = vfio_ap_matrix_dev_create(); >> + if (ret) >> + return ret; >> + >> + memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv)); >> + vfio_ap_drv.probe = vfio_ap_queue_dev_probe; >> + vfio_ap_drv.ids = ap_queue_ids; >> + >> + ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, >> VFIO_AP_DRV_NAME); >> + if (ret) { >> + vfio_ap_matrix_dev_destroy(ap_matrix); >> + return ret; >> + } >> + >> + return 0; >> +} >> + >> +void __exit vfio_ap_exit(void) >> +{ >> + ap_driver_unregister(&vfio_ap_drv); >> + vfio_ap_matrix_dev_destroy(ap_matrix); >> +} >> + >> +module_init(vfio_ap_init); >> +module_exit(vfio_ap_exit); >> diff --git a/drivers/s390/crypto/vfio_ap_private.h >> b/drivers/s390/crypto/vfio_ap_private.h >> new file mode 100644 >> index 0000000..3505947 >> --- /dev/null >> +++ b/drivers/s390/crypto/vfio_ap_private.h >> @@ -0,0 +1,20 @@ >> +/* >> + * Private data and functions for adjunct processor VFIO matrix driver. >> + * >> + * Copyright IBM Corp. 2017 >> + * Author(s): Tony Krowiak >> + */ >> + >> +#ifndef _VFIO_AP_PRIVATE_H_ >> +#define _VFIO_AP_PRIVATE_H_ >> + >> +#include >> + >> +#define VFIO_AP_MODULE_NAME "vfio_ap" >> +#define VFIO_AP_DRV_NAME "vfio_ap" >> + >> +struct ap_matrix { >> + struct device device; >> +}; >> + >> +#endif /* _VFIO_AP_PRIVATE_H_ */ >> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h >> index e3301db..cf2a5e9 100644 >> --- a/include/uapi/linux/vfio.h >> +++ b/include/uapi/linux/vfio.h >> @@ -200,6 +200,7 @@ struct vfio_device_info { >> #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform >> device */ >> #define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */ >> #define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */ >> +#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */ >> __u32 num_regions; /* Max region index + 1 */ >> __u32 num_irqs; /* Max IRQ index + 1 */ >> }; >> @@ -215,6 +216,7 @@ struct vfio_device_info { >> #define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform" >> #define VFIO_DEVICE_API_AMBA_STRING "vfio-amba" >> #define VFIO_DEVICE_API_CCW_STRING "vfio-ccw" >> +#define VFIO_DEVICE_API_AP_STRING "vfio-ap" >> >> /** >> * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8, > >