Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp453936pxa; Tue, 4 Aug 2020 09:23:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzCf4JQvokkQvWXMXAorfgrUt9wIPmO6d0Dx9hdF1UrJa0DW8KO7Q6MsHSbF1TM6P09jORa X-Received: by 2002:a17:906:d057:: with SMTP id bo23mr22043396ejb.210.1596558229138; Tue, 04 Aug 2020 09:23:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1596558229; cv=none; d=google.com; s=arc-20160816; b=YWz7c1+sGcrV4915BypTUnQRCaD5iy/WJLM8pukK9aKFcJXgL5NfOsRFmSh8AEpHQR JhoZW2t+wgUGexK7JSa1PTBP+zkkLHi4fdyuyKyGUuMcLnegtquWAjlpF0bsDbH0SitO Qrv9VEkNBDZMVJVtGNGE+b0EmQxA+i1gh0kJ1wk2YNDYMqUAgNKZ99k/4eMSz+IjalDP bpDhLa1qKNDHcW7GXpwAlHnmEreeabWie+HfsCHWTxVmA3I3zRCvnMJE7cJ6XiM8/wHI rnOceCn+aynEPJFYE5BgED5cZ8pJ7PWOfFGveFdNf2Sq9MrZQPe4fkhj+BSnC1KOJTnm 7VkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=PjtritRhIRwY+cn+BqykJaGzBMtkOtCDgHhypQ9ApCs=; b=BMXtW5hp2XLXLKsMAmaflEyVCphKnanmvy79THxfvdlb3AsHDKP73wS2pUJt4dVVU9 pCQGNHazAIAdEpbjxvKznED6gs+oO3sf5Aezreq6+AW4HB80CNKQtR+Onuxw/nux4GrP yXWD4d0VL9TOTxlivi0vW5cFg4+Xx92dSMaSe9eYkoAZxmi6waCSXoI9ZUqQUqkUFzs5 11YnH97Lff6iGR8y3wia3Zehj8eKjIiHHTauOzSMU0nVJ/nj5zDC4TNMZGfYyZkzw7fk AgIZ5Sh5t8frq8n1NqtESpGTYyU/olMvKuStLGXHvdEXi33moP/4y/L6u3pXYQt5wdB7 iTjw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mellanox.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id pg18si1693084ejb.614.2020.08.04.09.23.24; Tue, 04 Aug 2020 09:23:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mellanox.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729712AbgHDQU6 (ORCPT + 99 others); Tue, 4 Aug 2020 12:20:58 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:43838 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726011AbgHDQUz (ORCPT ); Tue, 4 Aug 2020 12:20:55 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eli@mellanox.com) with SMTP; 4 Aug 2020 19:20:51 +0300 Received: from nps-server-21.mtl.labs.mlnx (nps-server-21.mtl.labs.mlnx [10.237.240.120]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 074GKpau023986; Tue, 4 Aug 2020 19:20:51 +0300 Received: from nps-server-21.mtl.labs.mlnx (localhost [127.0.0.1]) by nps-server-21.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id 074GKpCL022632; Tue, 4 Aug 2020 19:20:51 +0300 Received: (from eli@localhost) by nps-server-21.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id 074GKpkj022631; Tue, 4 Aug 2020 19:20:51 +0300 From: Eli Cohen To: mst@redhat.com, jasowang@redhat.com, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Cc: shahafs@mellanox.com, saeedm@mellanox.com, parav@mellanox.com, Eli Cohen Subject: [PATCH V4 linux-next 00/12] VDPA support for Mellanox ConnectX devices Date: Tue, 4 Aug 2020 19:20:36 +0300 Message-Id: <20200804162048.22587-1-eli@mellanox.com> X-Mailer: git-send-email 2.26.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Michael, please note that this series depends on mlx5 core device driver patches in mlx5-next branch in git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git. git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301 Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok. The following series of patches provide VDPA support for Mellanox devices. The supported devices are ConnectX6 DX and newer. Currently, only a network driver is implemented; future patches will introduce a block device driver. iperf performance on a single queue is around 12 Gbps. Future patches will introduce multi queue support. The files are organized in such a way that code that can be used by different VDPA implementations will be placed in a common are resides in drivers/vdpa/mlx5/core. Only virtual functions are currently supported. Also, certain firmware capabilities must be set to enable the driver. Physical functions (PFs) are skipped by the driver. To make use of the VDPA net driver, one must load mlx5_vdpa. In such case, VFs will be operated by the VDPA driver. Although one can see a regular instance of a network driver on the VF, the VDPA driver takes precedence over the NIC driver, steering-wize. Currently, the device/interface infrastructure in mlx5_core is used to probe drivers. Future patches will introduce virtbus as a means to register devices and drivers and VDPA will be adapted to it. The mlx5 mode of operation required to support VDPA is switchdev mode. Once can use Linux or OVS bridge to take care of layer 2 switching. In order to provide virtio networking to a guest, an updated version of qemu is required. This version has been tested by the following quemu version: url: https://github.com/jasowang/qemu.git branch: vdpa Commit ID: 6f4e59b807db V2->V3 Fix makefile to use include path relative to the root of the kernel V3-V4 Rebase Jason's patches on linux-next branch Fix krobot error on mips arch Make use of the free callback to destroy resoruces on unload Use VIRTIO_F_ACCESS_PLATFORM instead of legacy VIRTIO_F_IOMMU_PLATFORM Add empty implementations for get_vq_notification() and get_vq_irq() Eli Cohen (6): net/vdpa: Use struct for set/get vq state vdpa: Modify get_vq_state() to return error code vdpa/mlx5: Add hardware descriptive header file vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add VDPA driver for supported mlx5 devices Jason Wang (5): vhost-vdpa: refine ioctl pre-processing vhost: generialize backend features setting/getting vhost-vdpa: support get/set backend features vhost-vdpa: support IOTLB batching hints vdpasim: support batch updating Max Gurtovoy (1): vdpa: remove hard coded virtq num drivers/vdpa/Kconfig | 19 + drivers/vdpa/Makefile | 1 + drivers/vdpa/ifcvf/ifcvf_base.c | 4 +- drivers/vdpa/ifcvf/ifcvf_base.h | 4 +- drivers/vdpa/ifcvf/ifcvf_main.c | 13 +- drivers/vdpa/mlx5/Makefile | 4 + drivers/vdpa/mlx5/core/mlx5_vdpa.h | 91 ++ drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h | 168 ++ drivers/vdpa/mlx5/core/mr.c | 484 ++++++ drivers/vdpa/mlx5/core/resources.c | 284 ++++ drivers/vdpa/mlx5/net/main.c | 76 + drivers/vdpa/mlx5/net/mlx5_vnet.c | 1965 ++++++++++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.h | 24 + drivers/vdpa/vdpa.c | 3 + drivers/vdpa/vdpa_sim/vdpa_sim.c | 53 +- drivers/vhost/net.c | 18 +- drivers/vhost/vdpa.c | 76 +- drivers/vhost/vhost.c | 15 + drivers/vhost/vhost.h | 2 + include/linux/vdpa.h | 24 +- include/uapi/linux/vhost.h | 2 + include/uapi/linux/vhost_types.h | 11 + 22 files changed, 3284 insertions(+), 57 deletions(-) create mode 100644 drivers/vdpa/mlx5/Makefile create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h create mode 100644 drivers/vdpa/mlx5/core/mr.c create mode 100644 drivers/vdpa/mlx5/core/resources.c create mode 100644 drivers/vdpa/mlx5/net/main.c create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h -- 2.26.0