Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Matthew Rosato <mjrosato@linux.ibm.com>
To:     linux-s390@vger.kernel.org
Cc:     alex.williamson@redhat.com, cohuck@redhat.com,
        schnelle@linux.ibm.com, farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        svens@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
        imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, joro@8bytes.org, will@kernel.org,
        pbonzini@redhat.com, corbet@lwn.net, jgg@nvidia.com,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        iommu@lists.linux-foundation.org, linux-doc@vger.kernel.org
Subject: [PATCH v4 00/32] KVM: s390: enable zPCI for interpretive execution
Date:   Mon, 14 Mar 2022 15:44:19 -0400
Message-Id: <20220314194451.58266-1-mjrosato@linux.ibm.com>
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
Precedence: bulk

Note: A few patches in this series are dependent on Baolu's IOMMU domain ops    
split, which is currently in the next branch of linux-iommu. This series        
applies on top:                                                                 
https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git     

Enable interpretive execution of zPCI instructions + adapter interruption
forwarding for s390x KVM vfio-pci.  This is done by introducing a new IOMMU
domain for s390x (KVM-managed), indicating via vfio that this IOMMU domain
should be used instead of the default, with subsequent management of the
hardware assists being handled via a new KVM ioctl for zPCI management.

By allowing intepretation of zPCI instructions and firmware delivery of         
interrupts to guests, we can significantly reduce the frequency of guest        
SIE exits for zPCI.  We then see additional gains by handling a hot-path        
instruction that can still intercept to the hypervisor (RPCIT) directly         
in kvm via the new IOMMU domain, whose map operations update the host           
DMA table with pinned guest entries over the specified range.

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by
default if available in kernel+qemu.

Will reply with a link to the associated QEMU series.

Changelog v3->v4:
v3: https://lore.kernel.org/kvm/20220204211536.321475-1-mjrosato@linux.ibm.com/ 
- Significant overhaul of the userspace API.  Remove all vfio device
  feature ioctls.  Remove CONFIG_VFIO_PCI_ZDEV, this is once again always
  built with vfio-pci for s390; IS_ENABLED checks can instead look at
  CONFIG_VFIO_PCI.  Most earlier patches in the series could maintain
  their reviews, but some needed to be removed due to required code
  changes.
- Instead use a KVM ioctl for zPCI management.  The API is very similar
  to the feature ioctls used in the prior series, with an additional step
  to create an association between an iommu domain + KVM + zPCI device.
- Introduce a new iommu domain ops type for s390-iommu, to be used when         
  KVM manages the IOMMU instead of in response to VFIO mapping ioctls 
- Add a iommu method for specifying the type of domain to allocate
- Add a new type to vfio_iommu_type1 (KVM-owned) to trigger the allocation
  of the KVM-owned IOMMU domain when zPCI interpretation is requested.
  In this case, the KVM-owned type is specified on VFIO_SET_IOMMU. 
- Wire the RPCIT intercepts into the new IOMMU domain via the kernel
  IOMMU API 
- Remove a bunch of unnecessary symbol externs, make the associated
  functions static
- Now that we keep a list of zPCI associated with a given KVM, we can do
  fh lookup on this list vs the list of all zPCI on the host.  We only
  need to do a host-wide fh lookup during the initial device<->KVM
  association.


Matthew Rosato (32):
  s390/sclp: detect the zPCI load/store interpretation facility
  s390/sclp: detect the AISII facility
  s390/sclp: detect the AENI facility
  s390/sclp: detect the AISI facility
  s390/airq: pass more TPI info to airq handlers
  s390/airq: allow for airq structure that uses an input vector
  s390/pci: externalize the SIC operation controls and routine
  s390/pci: stash associated GISA designation
  s390/pci: export some routines related to RPCIT processing
  s390/pci: stash dtsm and maxstbl
  s390/pci: add helper function to find device by handle
  s390/pci: get SHM information from list pci
  s390/pci: return status from zpci_refresh_trans
  iommu: introduce iommu_domain_alloc_type and the KVM type
  vfio: introduce KVM-owned IOMMU type
  vfio-pci/zdev: add function handle to clp base capability
  KVM: s390: pci: add basic kvm_zdev structure
  iommu/s390: add support for IOMMU_DOMAIN_KVM
  KVM: s390: pci: do initial setup for AEN interpretation
  KVM: s390: pci: enable host forwarding of Adapter Event Notifications
  KVM: s390: mechanism to enable guest zPCI Interpretation
  KVM: s390: pci: routines for (dis)associating zPCI devices with a KVM
  KVM: s390: pci: provide routines for enabling/disabling interpretation
  KVM: s390: pci: provide routines for enabling/disabling interrupt
    forwarding
  KVM: s390: pci: provide routines for enabling/disabling IOAT assist
  KVM: s390: pci: handle refresh of PCI translations
  KVM: s390: intercept the rpcit instruction
  KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
  vfio-pci/zdev: add DTSM to clp group capability
  KVM: s390: introduce CPU feature for zPCI Interpretation
  MAINTAINERS: additional files related kvm s390 pci passthrough
  MAINTAINERS: update s390 IOMMU entry

 Documentation/virt/kvm/api.rst   |  60 +++
 MAINTAINERS                      |   4 +-
 arch/s390/include/asm/airq.h     |   7 +-
 arch/s390/include/asm/kvm_host.h |   7 +
 arch/s390/include/asm/kvm_pci.h  |  40 ++
 arch/s390/include/asm/pci.h      |  12 +
 arch/s390/include/asm/pci_clp.h  |  11 +-
 arch/s390/include/asm/pci_dma.h  |   3 +
 arch/s390/include/asm/pci_insn.h |  31 +-
 arch/s390/include/asm/sclp.h     |   4 +
 arch/s390/include/asm/tpi.h      |  13 +
 arch/s390/include/uapi/asm/kvm.h |   1 +
 arch/s390/kvm/Makefile           |   1 +
 arch/s390/kvm/interrupt.c        |  95 +++-
 arch/s390/kvm/kvm-s390.c         |  90 +++-
 arch/s390/kvm/kvm-s390.h         |  10 +
 arch/s390/kvm/pci.c              | 833 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |  63 +++
 arch/s390/kvm/priv.c             |  46 ++
 arch/s390/pci/pci.c              |  31 ++
 arch/s390/pci/pci_clp.c          |  28 +-
 arch/s390/pci/pci_dma.c          |   7 +-
 arch/s390/pci/pci_insn.c         |  15 +-
 arch/s390/pci/pci_irq.c          |  48 +-
 drivers/iommu/Kconfig            |   8 +
 drivers/iommu/Makefile           |   1 +
 drivers/iommu/iommu.c            |   7 +
 drivers/iommu/s390-iommu.c       |  53 +-
 drivers/iommu/s390-iommu.h       |  53 ++
 drivers/iommu/s390-kvm-iommu.c   | 469 +++++++++++++++++
 drivers/s390/char/sclp_early.c   |   4 +
 drivers/s390/cio/airq.c          |  12 +-
 drivers/s390/cio/qdio_thinint.c  |   6 +-
 drivers/s390/crypto/ap_bus.c     |   9 +-
 drivers/s390/virtio/virtio_ccw.c |   6 +-
 drivers/vfio/pci/vfio_pci_zdev.c |  17 +-
 drivers/vfio/vfio_iommu_type1.c  |  12 +-
 include/linux/iommu.h            |  12 +
 include/uapi/linux/kvm.h         |  43 ++
 include/uapi/linux/vfio.h        |   6 +
 include/uapi/linux/vfio_zdev.h   |   6 +
 41 files changed, 2090 insertions(+), 94 deletions(-)
 create mode 100644 arch/s390/include/asm/kvm_pci.h
 create mode 100644 arch/s390/kvm/pci.c
 create mode 100644 arch/s390/kvm/pci.h
 create mode 100644 drivers/iommu/s390-iommu.h
 create mode 100644 drivers/iommu/s390-kvm-iommu.c

-- 
2.27.0