Received: by 2002:ac0:a5b6:0:0:0:0:0 with SMTP id m51-v6csp3808319imm; Mon, 4 Jun 2018 09:33:33 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL9cp5nQQ4lOtjUa5Bdad12AL2S5KcqfxrMu5XSC17tORVlzXAKry4HhUs1H+8fGSNSN51s X-Received: by 2002:a17:902:7009:: with SMTP id y9-v6mr22727516plk.217.1528130013715; Mon, 04 Jun 2018 09:33:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528130013; cv=none; d=google.com; s=arc-20160816; b=jpIjG+46O0uxyx16r8c927VTEcdmn2Wl66KRTvwUCU4CyT3s/Blc/ONh1SHhHfDYOo +TIwZo//z2GDHsW91/yd+MlUWtdoHRVr070GM39dOZdqvqjN5OJqHvID1eM1BuGwCJ4t ieejun4g/030owy1J+GVNoKwa5MZU/KuAReknrW1DBHYSkiY84EQgpwzO7GwO7qafl2O fHskJx8c1fEIzn7ZxL5gDTtOnA9CfQOi/lP0ST/TbFax8+Ages2m9j1A46vVX93w2Y07 g9JjXpIJ3pnI8QmWCBkPyE6lOnHVci3iy0SWAc6ZpYr080PZTFYYtUInoy7MDIKlDJ9g DlRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :arc-authentication-results; bh=gSg4E/nMf+Zfslcq33RKCbYOmlwb38ihd61WHQo1dxQ=; b=MXzKBdEx8lSuWqJpcqyiajXY94NxD67ecUOaedxr9o4nDKR6f+hs204Qa9p3dmrOjt aqvEizVj54NSLIjp3pqzz8MCQdEeAYcjIGUgqB5fkgQ6Tg64dksF6U08j/u3QiFRO8u5 YmeC0nXiFegkkOWV6xSXU/z9DlGFYFMSn8w1U5oQ+hhbVyd/BYiQ51FzJZGDDQa6tufX EDVuuFZPHnnRwpdx4XUaVpsOyNapBBr7JRA3J0v1R4KD4XO83wnkH+H0oCUCoRLI7yVU Iod5oMMYQctvSrpmMSL+qD4up51a1j1bANNyzJ3MQjysmog0bqKAKYZepZ61L6W5b4Pa Rw9g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u13-v6si10714885plm.99.2018.06.04.09.33.18; Mon, 04 Jun 2018 09:33:33 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751487AbeFDQcg (ORCPT + 99 others); Mon, 4 Jun 2018 12:32:36 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:40428 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751093AbeFDQcb (ORCPT ); Mon, 4 Jun 2018 12:32:31 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 772DF407538E; Mon, 4 Jun 2018 16:32:30 +0000 (UTC) Received: from redhat.com (ovpn-120-211.rdu2.redhat.com [10.10.120.211]) by smtp.corp.redhat.com (Postfix) with SMTP id 5A95D10EE6C0; Mon, 4 Jun 2018 16:32:26 +0000 (UTC) Date: Mon, 4 Jun 2018 19:32:25 +0300 From: "Michael S. Tsirkin" To: Tiwei Bie Cc: bhelgaas@google.com, stefanha@redhat.com, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, virtio-dev@lists.oasis-open.org, linux-pci@vger.kernel.org, dan.daly@intel.com, mark.d.rustad@intel.com, alexander.h.duyck@intel.com, cunming.liang@intel.com, zhihong.wang@intel.com Subject: Re: [PATCH v3] virtio_pci: support enabling VFs Message-ID: <20180604192222-mutt-send-email-mst@kernel.org> References: <20180601040239.1151-1-tiwei.bie@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180601040239.1151-1-tiwei.bie@intel.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 04 Jun 2018 16:32:30 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 04 Jun 2018 16:32:30 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'mst@redhat.com' RCPT:'' Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 01, 2018 at 12:02:39PM +0800, Tiwei Bie wrote: > There is a new feature bit allocated in virtio spec to > support SR-IOV (Single Root I/O Virtualization): > > https://github.com/oasis-tcs/virtio-spec/issues/11 > > This patch enables the support for this feature bit in > virtio driver. > > Signed-off-by: Tiwei Bie > --- OK but what about freeze/restore functions? I also wonder about kexec - virtio.c currently does: /* We always start by resetting the device, in case a previous * driver messed it up. This also tests that code path a little. */ dev->config->reset(dev); Do we need to do something like this for sriov? I also wonder whether PCI core should disable sriov for us. I wish there was a patch emulating this without vDPA for QEMU, would make it easy to test your patches. Do you happen to have something like this? Thanks, > v3: > - Drop the acks; > > v2: > - Disable VFs when unbinding the driver (Alex, MST); > - Don't use pci_sriov_configure_simple (Alex); > > drivers/virtio/virtio_pci_common.c | 30 ++++++++++++++++++++++++++++++ > drivers/virtio/virtio_pci_modern.c | 14 ++++++++++++++ > include/uapi/linux/virtio_config.h | 7 ++++++- > 3 files changed, 50 insertions(+), 1 deletion(-) > > diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c > index 48d4d1cf1cb6..1d4467b2dc31 100644 > --- a/drivers/virtio/virtio_pci_common.c > +++ b/drivers/virtio/virtio_pci_common.c > @@ -577,6 +577,8 @@ static void virtio_pci_remove(struct pci_dev *pci_dev) > struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev); > struct device *dev = get_device(&vp_dev->vdev.dev); > > + pci_disable_sriov(pci_dev); > + > unregister_virtio_device(&vp_dev->vdev); > > if (vp_dev->ioaddr) > @@ -588,6 +590,33 @@ static void virtio_pci_remove(struct pci_dev *pci_dev) > put_device(dev); > } > > +static int virtio_pci_sriov_configure(struct pci_dev *pci_dev, int num_vfs) > +{ > + struct virtio_pci_device *vp_dev = pci_get_drvdata(pci_dev); > + struct virtio_device *vdev = &vp_dev->vdev; > + int ret; > + > + if (!(vdev->config->get_status(vdev) & VIRTIO_CONFIG_S_DRIVER_OK)) > + return -EBUSY; > + > + if (!__virtio_test_bit(vdev, VIRTIO_F_SR_IOV)) > + return -EINVAL; > + > + if (pci_vfs_assigned(pci_dev)) > + return -EPERM; > + > + if (num_vfs == 0) { > + pci_disable_sriov(pci_dev); > + return 0; > + } > + > + ret = pci_enable_sriov(pci_dev, num_vfs); > + if (ret < 0) > + return ret; > + > + return num_vfs; > +} > + > static struct pci_driver virtio_pci_driver = { > .name = "virtio-pci", > .id_table = virtio_pci_id_table, > @@ -596,6 +625,7 @@ static struct pci_driver virtio_pci_driver = { > #ifdef CONFIG_PM_SLEEP > .driver.pm = &virtio_pci_pm_ops, > #endif > + .sriov_configure = virtio_pci_sriov_configure, > }; > > module_pci_driver(virtio_pci_driver); > diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c > index 2555d80f6eec..07571daccfec 100644 > --- a/drivers/virtio/virtio_pci_modern.c > +++ b/drivers/virtio/virtio_pci_modern.c > @@ -153,14 +153,28 @@ static u64 vp_get_features(struct virtio_device *vdev) > return features; > } > > +static void vp_transport_features(struct virtio_device *vdev, u64 features) > +{ > + struct virtio_pci_device *vp_dev = to_vp_device(vdev); > + struct pci_dev *pci_dev = vp_dev->pci_dev; > + > + if ((features & BIT_ULL(VIRTIO_F_SR_IOV)) && > + pci_find_ext_capability(pci_dev, PCI_EXT_CAP_ID_SRIOV)) > + __virtio_set_bit(vdev, VIRTIO_F_SR_IOV); > +} > + > /* virtio config->finalize_features() implementation */ > static int vp_finalize_features(struct virtio_device *vdev) > { > struct virtio_pci_device *vp_dev = to_vp_device(vdev); > + u64 features = vdev->features; > > /* Give virtio_ring a chance to accept features. */ > vring_transport_features(vdev); > > + /* Give virtio_pci a chance to accept features. */ > + vp_transport_features(vdev, features); > + > if (!__virtio_test_bit(vdev, VIRTIO_F_VERSION_1)) { > dev_err(&vdev->dev, "virtio: device uses modern interface " > "but does not have VIRTIO_F_VERSION_1\n"); > diff --git a/include/uapi/linux/virtio_config.h b/include/uapi/linux/virtio_config.h > index 308e2096291f..b7c1f4e7d59e 100644 > --- a/include/uapi/linux/virtio_config.h > +++ b/include/uapi/linux/virtio_config.h > @@ -49,7 +49,7 @@ > * transport being used (eg. virtio_ring), the rest are per-device feature > * bits. */ > #define VIRTIO_TRANSPORT_F_START 28 > -#define VIRTIO_TRANSPORT_F_END 34 > +#define VIRTIO_TRANSPORT_F_END 38 > > #ifndef VIRTIO_CONFIG_NO_LEGACY > /* Do we get callbacks when the ring is completely used, even if we've > @@ -71,4 +71,9 @@ > * this is for compatibility with legacy systems. > */ > #define VIRTIO_F_IOMMU_PLATFORM 33 > + > +/* > + * Does the device support Single Root I/O Virtualization? > + */ > +#define VIRTIO_F_SR_IOV 37 > #endif /* _UAPI_LINUX_VIRTIO_CONFIG_H */ > -- > 2.17.0