Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Matthew Rosato <mjrosato@linux.ibm.com>
To:     linux-s390@vger.kernel.org
Cc:     alex.williamson@redhat.com, cohuck@redhat.com,
        schnelle@linux.ibm.com, farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        svens@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
        imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, pbonzini@redhat.com, corbet@lwn.net,
        jgg@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-doc@vger.kernel.org
Subject: [PATCH v7 00/22] KVM: s390: enable zPCI for interpretive execution
Date:   Fri, 13 May 2022 15:14:47 -0400
Message-Id: <20220513191509.272897-1-mjrosato@linux.ibm.com>
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
Precedence: bulk

Enable interpretive execution of zPCI instructions + adapter interruption
forwarding for s390x KVM vfio-pci.  This is done by triggering a routine
when the VFIO group is associated with the KVM guest, transmitting to
firmware a special token (GISA designation) to enable that specific guest
for interpretive execution on that zPCI device.  Load/store interpreation
enablement is then controlled by userspace (based upon whether or not a
SHM bit is placed in the virtual function handle).  Adapter Event
Notification interpretation is controlled from userspace via a new KVM
ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of
interrupts to guests, we can reduce the frequency of guest SIE exits for
zPCI.  

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by
default if available in kernel+qemu.

Some minor changes are already suggested for the QEMU series, but for
now the v5 one is still compatible (with one exception: in
linux-headers/linux/kvm.h KVM_CAP_S390_ZPCI_OP is now changed to 216)

https://lore.kernel.org/kvm/20220404181726.60291-1-mjrosato@linux.ibm.com/

Changelog v6->v7:
- Add some more R-b tags
- Mark a few routines static (kernel bot)
- Add a few lockdep_assert_held calls (Christian)
- Add a new condition under which locked_vm is defined (Christian)
- After discussions with Jason, let's go back to using a Kconfig option
  to control vfio-pci-zdev inclusion so we can avoid some symbol_get
  messiness.  The name was chosen so that we can re-introduce zdev
  for non-KVM environments in the future, should it be needed.
- Since we re-introduce a Kconfig option, we can go back to using this in
  various places instead of CONFIG_VFIO_PCI.
- Expose separate interfaces for register and unregister of kvm (Jason)
- Cleanup KVM association during zdev device_close; if the device fd is
  closed before we receive the kvm notifier, we will never see the
  notification because we unregister it, but cleanup must still be done.
  Conversely, if the kvm fd closes before the device is closed, we
  should get the NULL notifier from kvm_destroy_devices since the notifier
  has not yet been unregistered.
- As a result of the above, we no longer need to clear the pci list during
  vm exit.  But to be safe let's WARN_ON_ONCE if it's found to be non-empty
  during free_vm.
- Add kzdev_lock to protect the kzdev
- uapi KVM_CAP_S390_ZPCI_OP moved to 216 (previously 214)

Matthew Rosato (22):
  s390/sclp: detect the zPCI load/store interpretation facility
  s390/sclp: detect the AISII facility
  s390/sclp: detect the AENI facility
  s390/sclp: detect the AISI facility
  s390/airq: pass more TPI info to airq handlers
  s390/airq: allow for airq structure that uses an input vector
  s390/pci: externalize the SIC operation controls and routine
  s390/pci: stash associated GISA designation
  s390/pci: stash dtsm and maxstbl
  vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM
  KVM: s390: pci: add basic kvm_zdev structure
  KVM: s390: pci: do initial setup for AEN interpretation
  KVM: s390: pci: enable host forwarding of Adapter Event Notifications
  KVM: s390: mechanism to enable guest zPCI Interpretation
  KVM: s390: pci: provide routines for enabling/disabling interrupt
    forwarding
  KVM: s390: pci: add routines to start/stop interpretive execution
  vfio-pci/zdev: add open/close device hooks
  vfio-pci/zdev: add function handle to clp base capability
  vfio-pci/zdev: different maxstbl for interpreted devices
  KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
  KVM: s390: introduce CPU feature for zPCI Interpretation
  MAINTAINERS: additional files related kvm s390 pci passthrough

 Documentation/virt/kvm/api.rst   |  45 ++
 MAINTAINERS                      |   1 +
 arch/s390/include/asm/airq.h     |   7 +-
 arch/s390/include/asm/kvm_host.h |  26 ++
 arch/s390/include/asm/pci.h      |  13 +
 arch/s390/include/asm/pci_clp.h  |   9 +-
 arch/s390/include/asm/pci_insn.h |  29 +-
 arch/s390/include/asm/sclp.h     |   4 +
 arch/s390/include/asm/tpi.h      |  13 +
 arch/s390/include/uapi/asm/kvm.h |   1 +
 arch/s390/kvm/Makefile           |   1 +
 arch/s390/kvm/interrupt.c        |  96 ++++-
 arch/s390/kvm/kvm-s390.c         |  94 ++++-
 arch/s390/kvm/kvm-s390.h         |  10 +
 arch/s390/kvm/pci.c              | 686 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |  88 ++++
 arch/s390/pci/pci.c              |  16 +
 arch/s390/pci/pci_clp.c          |   7 +
 arch/s390/pci/pci_insn.c         |   4 +-
 arch/s390/pci/pci_irq.c          |  48 ++-
 drivers/s390/char/sclp_early.c   |   4 +
 drivers/s390/cio/airq.c          |  12 +-
 drivers/s390/cio/qdio_thinint.c  |   6 +-
 drivers/s390/crypto/ap_bus.c     |   9 +-
 drivers/s390/virtio/virtio_ccw.c |   6 +-
 drivers/vfio/pci/Kconfig         |  11 +
 drivers/vfio/pci/Makefile        |   2 +-
 drivers/vfio/pci/vfio_pci_core.c |   2 +
 drivers/vfio/pci/vfio_pci_zdev.c |  65 ++-
 include/linux/sched/user.h       |   3 +-
 include/linux/vfio_pci_core.h    |  12 +-
 include/uapi/linux/kvm.h         |  31 ++
 include/uapi/linux/vfio_zdev.h   |   7 +
 33 files changed, 1313 insertions(+), 55 deletions(-)
 create mode 100644 arch/s390/kvm/pci.c
 create mode 100644 arch/s390/kvm/pci.h

-- 
2.27.0