Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.35 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.35; helo=mail.nvidia.com;
From:   Max Gurtovoy <mgurtovoy@nvidia.com>
To:     <jgg@nvidia.com>, <alex.williamson@redhat.com>,
        <cohuck@redhat.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     <liranl@nvidia.com>, <oren@nvidia.com>, <tzahio@nvidia.com>,
        <leonro@nvidia.com>, <yarong@nvidia.com>, <aviadye@nvidia.com>,
        <shahafs@nvidia.com>, <artemp@nvidia.com>, <kwankhede@nvidia.com>,
        <ACurrid@nvidia.com>, <cjia@nvidia.com>, <yishaih@nvidia.com>,
        <mjrosato@linux.ibm.com>, <aik@ozlabs.ru>, <hch@lst.de>,
        Max Gurtovoy <mgurtovoy@nvidia.com>
Subject: [PATCH v3 0/9] Introduce vfio-pci-core subsystem
Date:   Tue, 9 Mar 2021 08:33:48 +0000
Message-ID: <20210309083357.65467-1-mgurtovoy@nvidia.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2021 08:34:05.2343
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 189f57ac-8b41-41b7-9ee6-08d8e2d6199b
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.35];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT067.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4641
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Alex and Cornelia,

This series split the vfio_pci driver into 2 parts: pci drivers and a
subsystem driver that will also be library of code. The main pci driver,
vfio_pci.ko will be used as before and it will bind to the subsystem
driver vfio_pci_core.ko to register to the VFIO subsystem.
New vendor vfio pci drivers were introduced in this series are:
- igd_vfio_pci.ko
- nvlink2gpu_vfio_pci.ko
- npu2_vfio_pci.ko

These drivers also will bind to the subsystem driver vfio_pci_core.ko to
register to the VFIO subsystem. These drivers will also add vendor
specific extensions that are relevant only for the devices that will be
driven by them. This is a typical Linux subsystem framework behaviour.

This series is coming to solve some of the issues that were raised in the
previous attempt for extending vfio-pci for vendor specific
functionality: https://lkml.org/lkml/2020/5/17/376 by Yan Zhao.

This solution is also deterministic in a sense that when a user will
bind to a vendor specific vfio-pci driver, it will get all the special
goodies of the HW. Non-common code will be pushed only to the specific
vfio_pci driver.
 
This subsystem framework will also ease on adding new vendor specific
functionality to VFIO devices in the future by allowing another module
to provide the pci_driver that can setup number of details before
registering to VFIO subsystem (such as inject its own operations).

Below we can see the proposed changes:

+-------------------------------------------------------------------+
|                                                                   |
|                               VFIO                                |
|                                                                   |
+-------------------------------------------------------------------+
                                                             
+-------------------------------------------------------------------+
|                                                                   |
|                           VFIO_PCI_CORE                           |
|                                                                   |
+-------------------------------------------------------------------+

+--------------+ +------------+ +-------------+ +-------------------+
|              | |            | |             | |                   |
|              | |            | |             | |                   |
|   VFIO_PCI   | |IGD_VFIO_PCI| |NPU2_VFIO_PCI| |NVLINK2GPU_VFIO_PCI|
|              | |            | |             | |                   |
|              | |            | |             | |                   |
+--------------+ +------------+ +-------------+ +-------------------+

Patches (1/9) - (4/9) introduce the above changes for vfio_pci and
vfio_pci_core.

Patches (6/9) and (7/9) are a preparation for adding nvlink2 related vfio
pci drivers.

Patch (8/9) introduce new npu2_vfio_pci and nvlink2gpu_vfio_pci drivers
that will drive NVLINK2 capable devices exist today (IBMs emulated PCI
managment device and NVIDIAs NVLINK2 capable GPUs). These drivers add
vendor specific functionality that is related to the IBM NPU2 unit which
is an NVLink2 host bus adapter and to capable NVIDIA GPUs.

Patch (9/9) introduce new igd_vfio_pci driver for adding special extensions
for INTELs Graphics card (GVT-d).

All new 3 vendor specific vfio_pci drivers can be extended easily and
new vendor specific functionality might be added to the needed vendor
drivers. In case there is a vendor specific vfio_pci driver for the
device that a user would like to use, this driver should be bounded to
that device. Otherwise, vfio_pci.ko should be bounded to it (backward
compatability is also supported).

This framework will allow adding more HW specific features such as Live
Migration in the future by extending existing vendor vfio_pci drivers or
creating new ones (e.g. mlx5_vfio_pci.ko that will drive live migration
for mlx5 PCI devices and mlx5_snap_vfio_pci.ko that will drive live
migration for mlx5_snap devices such as mlx5 NVMe and Virtio-BLK SNAP
devices).

Testing:
1. vfio_pci.ko was tested by using 2 VirtIO-BLK physical functions based
   on NVIDIAs Bluefield-2 SNAP technology. These 2 PCI functions were
   passed to a QEMU based VM after binding to vfio_pci.ko and basic FIO
   read/write was issued in the VM to each exposed block device (vda, vdb).
2. igd_vfio_pci.ko was only compiled and loaded/unloaded successfully on x86 server.
3. npu2_vfio_pci.ko and nvlink2gpu_vfio_pci.KO were successfully
   compiled and loaded/unloaded on P9 server + vfio probe/remove of devices (without
   QEMU/VM).

Note: After this series will be approved, a new discovery/matching mechanism
      should be introduced in order to help users to decide which driver should
      be bounded to which device.

changes from v2:
 - Use container_of approach between core and vfio_pci drivers.
 - Comment on private/public attributes for the core structure to ease on
   code maintanance.
 - rename core structure to vfio_pci_core_device.
 - create 3 new vendor drivers (igd_vfio_pci, npu2_vfio_pci,
   nvlink2gpu_vfio_pci) and preserve backward compatibility.
 - rebase on top of Linux 5.11 tag.
 - remove patches that were accepted as stand-alone.
 - removed mlx5_vfio_pci driver from this series (3 vendor drivers will be enough
   for emphasizing the approch).

changes from v1:
 - Create a private and public vfio-pci structures (From Alex)
 - register to vfio core directly from vfio-pci-core (for now, expose
   minimal public funcionality to vfio pci drivers. This will remove the
   implicit behaviour from v1. More power to the drivers can be added in
   the future)
 - Add patch 3/9 to emphasize the needed extension for LM feature (From
   Cornelia)
 - take/release refcount for the pci module during core open/release
 - update nvlink, igd and zdev to PowerNV, X86 and s390 extensions for
   vfio-pci core
 - fix segfault bugs in current vfio-pci zdev code

Max Gurtovoy (9):
  vfio-pci: rename vfio_pci.c to vfio_pci_core.c
  vfio-pci: rename vfio_pci_private.h to vfio_pci_core.h
  vfio-pci: rename vfio_pci_device to vfio_pci_core_device
  vfio-pci: introduce vfio_pci_core subsystem driver
  vfio/pci: introduce vfio_pci_device structure
  vfio-pci-core: export vfio_pci_register_dev_region function
  vfio/pci_core: split nvlink2 to nvlink2gpu and npu2
  vfio/pci: export nvlink2 support into vendor vfio_pci drivers
  vfio/pci: export igd support into vendor vfio_pci driver

 drivers/vfio/pci/Kconfig                      |   53 +-
 drivers/vfio/pci/Makefile                     |   20 +-
 .../pci/{vfio_pci_igd.c => igd_vfio_pci.c}    |  159 +-
 drivers/vfio/pci/igd_vfio_pci.h               |   24 +
 drivers/vfio/pci/npu2_trace.h                 |   50 +
 drivers/vfio/pci/npu2_vfio_pci.c              |  364 +++
 drivers/vfio/pci/npu2_vfio_pci.h              |   24 +
 .../vfio/pci/{trace.h => nvlink2gpu_trace.h}  |   27 +-
 ...io_pci_nvlink2.c => nvlink2gpu_vfio_pci.c} |  296 +-
 drivers/vfio/pci/nvlink2gpu_vfio_pci.h        |   24 +
 drivers/vfio/pci/vfio_pci.c                   | 2433 ++---------------
 drivers/vfio/pci/vfio_pci_config.c            |   70 +-
 drivers/vfio/pci/vfio_pci_core.c              | 2245 +++++++++++++++
 drivers/vfio/pci/vfio_pci_core.h              |  242 ++
 drivers/vfio/pci/vfio_pci_intrs.c             |   42 +-
 drivers/vfio/pci/vfio_pci_private.h           |  228 --
 drivers/vfio/pci/vfio_pci_rdwr.c              |   18 +-
 drivers/vfio/pci/vfio_pci_zdev.c              |   12 +-
 18 files changed, 3528 insertions(+), 2803 deletions(-)
 rename drivers/vfio/pci/{vfio_pci_igd.c => igd_vfio_pci.c} (58%)
 create mode 100644 drivers/vfio/pci/igd_vfio_pci.h
 create mode 100644 drivers/vfio/pci/npu2_trace.h
 create mode 100644 drivers/vfio/pci/npu2_vfio_pci.c
 create mode 100644 drivers/vfio/pci/npu2_vfio_pci.h
 rename drivers/vfio/pci/{trace.h => nvlink2gpu_trace.h} (72%)
 rename drivers/vfio/pci/{vfio_pci_nvlink2.c => nvlink2gpu_vfio_pci.c} (57%)
 create mode 100644 drivers/vfio/pci/nvlink2gpu_vfio_pci.h
 create mode 100644 drivers/vfio/pci/vfio_pci_core.c
 create mode 100644 drivers/vfio/pci/vfio_pci_core.h
 delete mode 100644 drivers/vfio/pci/vfio_pci_private.h

-- 
2.25.4