From: Tony Krowiak <[email protected]>
Several major objections were raised to design changes introduced in the v7
patch series, so in order to avoid an extended discussion around these
objections and to expedite acceptance of the series, the following changes
have been made for v8:
1. Removed the AP bus's ability to designate queues as 'used by host' or as
'used by alternate driver(s)'. The bind/unbind sysfs interfaces will be
used for managing the connection of AP queue devices to a zcrypt driver
or the VFIO AP driver.
2. Removed the 'activate' sysfs interfaces which allowed for over
provisioning of the mediated device as well as creation of mdevs with
overlapping matrixes. It was pointed out that both of these enhancements
break the mdev model. Consistency checking of the mdev matrix has
therefore been returned to the mediated matrix device's sysfs interfaces
for assigning adapters and domains:
* Verify that APQNs assigned to the mediated device are bound to the
VFIO AP device driver
* Verify that no APQN assigned to the mediated matrix device is assigned
to any other mediated matrix device.
4. Reworked the handling of the CRYCB in vSIE based upon patches introduced
by David in the mainline.
Notes:
=====
Patches 1-4 (by Harald) posted with this series are forthcoming via
Martins tree and are based on changes in the ap driver/bus that we use as a
foundation. They have been included here because some of the functions
in this patch series are dependent upon them.
Patches 5-6 (by David) are posted with this series because they are not
currently in our master branch. Patches 19 and 20 of this series are
dependent upon them. I believe David's patches are available in the
mainline now.
This patch series works with the v6 QEMU patches. There is no new QEMU
patchset version yet because there have been no review comments worthy of
creating a new series; only a couple of extremely minor nits.
Abstract:
========
On s390, we have cryptographic coprocessor cards, which are modeled on
Linux as devices on the AP bus. Each card can be partitioned into domains
which can be thought of as a set of hardware registers for processing
crypto commands. Crypto commands are sent to a specific domain within a
card is via a queue which is identified as a (card,domain) tuple. We model
this something like the following (assuming we have access to cards 3 and
4 and domains 1 and 2):
AP -> card3 -> queue (3,1)
-> queue (3,2)
-> card4 -> queue (4,1)
-> queue (4,2)
If we want to virtualize this, we can use a feature provided by the
hardware. We basically attach a satellite control block to our main
hardware virtualization control block and the hardware takes care of
most of the rest.
For this control block, we don't specify explicit tuples, but a list of
cards and a list of domains. The guest will get access to the cross
product.
Because of this, we need to take care that the lists provided to
different guests don't overlap; i.e., we need to enforce sane
configurations. Otherwise, one guest may get access to things like
secret keys for another guest.
The idea of this patch set is to introduce a new device, the matrix
device. This matrix device hangs off a different root and acts as the
parent node for mdev devices.
If you now want to give the tuples (4,1) and (4,2), you need to do the
following:
- Make sure the queues (4,1) and (4,2) belong to vfio_ap (see patches
#5 and #6)
- Create the mediated device.
- Assign card 4 and domains 1 and 2 to the mediated device
- Optionally activate the mediated device.
QEMU will now simply consume the mediated device and things should work.
For a complete description of the architecture and concepts underlying
the design, see the Documentation/s390/vfio-ap.txt file included with this
patch set.
v7 => v8 Change log:
===================
* Removed the AP bus's ability to designate queues as 'used by host' or as
'used by alternate driver(s)' and relying on bind/unbind for connecting
AP queue devices to a zcrypt driver or the VFIO AP driver.
* Removed 'activate' attribute from mediated device.
* Do consistency checking during device assignment:
1. Verify that APQNs assigned to the mediated device are bound to the
VFIO AP device driver
2. Verify that no APQN assigned to the mediated matrix device is assigned
to any other mediated matrix device.
* The attributes of a mediated matrix device that is in use by a guest can
not be changed - i.e., no device assignment/unassignment allowed
* A mediated matrix device that is in use by a guest can not be removed.
* Removed all printk logging from VFIO AP driver; allowing return codes
from interfaces to describe the error.
* Reworked the handling of the CRYCB in vSIE based upon patches introduced
by David in the mainline.
v6 => v7 Change log:
===================
* The AP bus gained the ability to designate queues as 'used by host'
or as 'used by alternate driver(s)'. This allows us to authorise access
(via the CRYCB) to queues that are not currently bound to the vfio_ap
driver. If a vfio_ap owned queue diss- and reapears it's guaranteed
to get bound back to the vfio_ap driver.
* The mediated device gained an 'activate' attribute. Sharing conflicts are
checked on activation now. If the device was not activated, the mdev
open still implies activation. An active ap_matrix_mdev device claims
it's resources -- an inactive does not.
* An active ap_matrix_mdev device can not be removed. An ap_matrix_mdev
that is hooked up with a guest can not be deactivated.
* An active ap_matrix_mdev device rejects assign_* and deassign_*
operations. Thus changing the CRYCB masks of a guest in order to
accomplys certain hotplug scenarios is planned, but not supported yet. In
previous versions it was possible to do those operations on a ap_matrix_mdev
that is hooked up to a guest, but the changes would take effect on the next
mdev_open.
* Synchronisation was reworked.
* The sysfs path of the parent device changed from /sys/devices/vfio_ap/matrix/
to /sys/devices/virtual/misc/vfio_ap/. The parent device is a misc
device now.
* The severity for most of the messages were reduced form error to
warning.
* We are not as thick headed about the zapq as we used to be in v6.
v5 => v6 Change log:
===================
* Added VSIE support - thanks to Pierre Morel
* Added VFIO_DEVICE_RESET ioctl
* Zeroizing AP queues when mediated device released and when
VFIO_DEVICE_RESET ioctl is invoked
* Removed /arch/s390/kvm/kvm-ap.c and arch/s390/include/asm/kvm-ap.h and
moved guest matrix configuration into vfio driver
* Removed temporary interfaces to be supplied by AP bus
* Made the variable that keeps track of mdev instance count an atomic_t
type
* Removed code iterating through vm_list to determine if another guest has
a queue .... not keep a list of matrix_mdev devices and verify against
that list. Removes the need for the kvm_lock.
* Added a sysfs attribute for the mediated matrix device to display the
matrix contained in the guest's CRYCB if a guest is using the mdev.
v4 => v5 Change log:
===================
* Verify AP queues bound to driver in mediated device open callback, prior
to configuring the matrix in the CRYCB
* Implement VFIO_DEVICE_RESET ioctl
* Zeroize queues on guest entry and exit
* Removed vnet from all email IBM email addresses referenced
* Add synchronization in mdev create/remove and open/release.
v4 => v5 Change log:
===================
* Added code to mdev open callback to ensure not more than one vfio-ap
device can be opened by a guest.
* Interpret AP instructions by default
* Removed patch implementing interface to enable/disable AP interpretation,
since that will now be done by default
* Removed patch to reset crypto attributes for ALL vcpus. That will be
submitted as a single patch since it will not be needed in this series -
i.e., it was called from the interface to enable/disable AP instructions
* All code for initializing crypto for a guest has been moved back to
kvm-s390.c, kvm_s390_crypto_init(kvm) function
* Maintaining a module reference count for the vfio_ap module so it is not
removed while a guest with AP devices is running.
v3 => v4 Change log:
===================
* Resolved issue with enabling ZCRYPT when KVM is enabled by using
#ifdef ZCRYPT in relevant functions
* Added patch with a new function for resetting the crypto attributes
for all vcpus to resolve the issue raised with running vcpus getting out
of sync.
* Removed KVM_S390_VM_CRYPTO_INTERPRET_AP: Setting interpretive exec mode
from vfio_ap driver when mdev device is opened.
v2 => v3 Change log:
===================
* Set APIE in VCPU setup function
* Renamed patch 13/15:
KVM: s390: Configure the guest's CRYCB
KVM: s390: Configure the guest's AP devices
* Fixed problem with building arch/s390/kvm/kvm-ap.c when CONFIG_ZCRYPT
not selected
* Removed patch introducing VSIE support for AP pending further
investigation
* Initialized AP maximum mask sizes - i.e., APM, AQM and ADM - from info
returned from PQAP(QCI) function
* Introduced a new device attribute to the KVM_S390_VM_CRYPTO attribute
group for setting a flag via the KVM_SET_DEVICE_ATTR ioctl to indicate
whether ECA_APIE should be set or not. The flag is used in the
kvm_s390_vcpu_crypto_setup() function to set ECA_APIE in the SIE block.
v1 => v2 Change log:
===================
* Added documentation vfio-ap.txt
* Renamed vfio_ap_matrix module and device driver to vfio_ap
* Use device core device list instead of maintaining list of matrix
devices in driver
* Added VSIE support for AP
* Create matrix device before registering VFIO AP device driver with the
AP bus
* Renamed the following files in drivers/s390/crypto:
* vfio_ap_matrix.drv -> vfio_ap_drv
* vfio_ap_matrix_private.h -> vfio_ap_private.h
* vfio_ap_matrix_ops.c -> vfio_ap_ops.c
* arch/s390/include/asm/kvm/ap-matrix-config.h
* Renamed to kvm-ap.h
* Changed the data type of the bit mask fields for the matrix structure
to unsigned long and create them with DECLARE_BITMAP
* Changed #define prefixes from AP_MATRIX to KVM_AP
* Changed function and structure prefixes from ap_matrix to kvm_ap
* Added function interface to check if AP Extended Addressing (APXA)
facility is installedCRYCB_FORMAT_MASK
* Added function interface to get the maximum ID for AP mask type
* Added function interface to set the AP execution mode
* arch/s390/kvm/ap-matrix-config.c
* Renamed to kvm-ap.c
* Changed function prefixes from ap_matrix to kvm_ap
* Added function to check if AP Extended Addressing (APXA) facility is
installed
* Added function to get the maximum ID for AP mask type
* Added function to set the AP execution mode
* Added a boolean parameter to the functions that retrieve the APM, AQM
and ADM bit mask fields from the CRYCB. If true, then the function
will clear the bits in the mask before returning a reference to it
* Added validation to verify that APM, AQM and ADM bits that are set do
not exceed the maximum ID value allowed
*
* arch/s390/include/asm/kvm_host.h
* Changed defined for ECA_AP to ECA_APIE - interpretive execution mode
* Added a flag to struct kvm_s390_crypto to indicate whether the
KVM_S390_VM_CPU_FEAT_AP CPU model feature for AP facilities is set
* Added two CPU facilities features to set STFLE.12 and STFLE.15
* arch/s390/kvm/kvm-s390.c
* Added initialization for new KVM_S390_VM_CPU_FEAT_AP CPU model feature
* Removed kvm_s390_apxa_installed() function
* Changed call to kvm_s390_apxa_installed() which has been removed to a
call to new kvm_ap_apxa_installed() function.
* Added code to kvm_s390_vcpu_crypto_setup() to set the new CPU model
feature flag in the kvm_s390_crypto structure
* Added CRYCB_FORMAT_MASK to mask CRYCBD
* arch/s390/tools/gen_facilities.c
* Added STFLE.12 and STFLE.15 to struct facility _def
* drivers/s390/crypto/vfio_ap_matrix_private.h
* Changed name of file to vfio_ap.private.h
* Changed #define prefixes from VFIO_AP_MATRIX to VFIO_AP
* struct ap_matrix: removed list fields and locks
* struct vfio_ap_queue: removed list field
* Renamed functions ap_matrix_mdev_register and ap_matrix_mdev_unregister
to vfio_ap_mdev_register and vfio_ap_mdev_unregister respectively
* drivers/s390/crypto/vfio_ap_matrix_drv.c
* Renamed file to drivers/s390/crypto/vfio_ap_drv.c
* Changed all #define, structure and function prefixes to vfio_ap
* probe function
* Changed root device name for the matrix device to vfio_ap:
i.e., /sys/devices/vfio_ap/matrix
* No longer storing the AP queue device in a list, it is retrievable via
the device core
* Removed unnecessary check whether matrix device exists
* Store the vfio_ap_queue structure in the private field of the ap_queue
structure rather than using list interface
* remove function
* Retrieve vfio_ap_queue structure from the struct ap_queue private
data rather than from a list
* Removed unnecesary check
* drivers/s390/crypto/vfio_ap_matrix_ops.c
* Renamed file to vfio_ap_ops.c
* Changed #define prefixes from AP_MATRIX to VFIO_AP
* Changed function name prefixes from ap_matrix to vfio_ap
* Removed ioctl to configure the CRYCB
* create function
* Removed ap_matrix_mdev_find_by_uuid() function - function is provided
by mdev core
* Removed available_instances verification, provided by mdev core
* Removed check to see if mediated device exists, handled by mdev core
* notifier function
* Configuring matrix here instead of via ioctl
* Set interpretive execution mode for all VCPUs
* Removed R/O attributes to display adapters and domains
* Added an R/O attribute to display the matrix
* assign_control_domain mdev attribute:
* Removed check to see if the domain is installed on the linux host
* Added check to verify the control domain ID does not exceed the max
value
* assign_adapter mdev attribute:
* Added check to verify the adapter ID does not exceed the max
value
* If any APQNs configured for the mediated matrix device that
have an APID matching the adapter ID being assigned are not
bound to the vfio_ap device driver then it is assumed that the APQN is
bound to another driver and assignment will fail
* assign_domain mdev attribute:
* Added check to verify the domain ID does not exceed the max
value
* If any APQNs configured for the mediated matrix device that
have an APQI matching the domain ID being assigned are not
bound to the vfio_ap device driver then it is assumed that the APQN is
bound to another driver and assignment will fail
* tools/arch/s390/include/uapi/asm/kvm.h
* removed KVM_S390_VM_CPU_FEAT_AP feature definition
David Hildenbrand (2):
KVM: s390: vsie: simulate VCPU SIE entry/exit
KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
Harald Freudenberger (4):
s390/zcrypt: Add ZAPQ inline function.
s390/zcrypt: Review inline assembler constraints.
s390/zcrypt: Show load of cards and queues in sysfs
s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h.
Pierre Morel (2):
KVM: s390: Clear Crypto Control Block when using vSIE
KVM: s390: Handling of Cypto control block in VSIE
Tony Krowiak (14):
KVM: s390: refactor crypto initialization
s390: vfio-ap: base implementation of VFIO AP device driver
s390: vfio-ap: register matrix device with VFIO mdev framework
s390: vfio-ap: sysfs interfaces to configure adapters
s390: vfio-ap: sysfs interfaces to configure domains
s390: vfio-ap: sysfs interfaces to configure control domains
s390: vfio-ap: sysfs interface to view matrix mdev matrix
KVM: s390: interfaces to clear CRYCB masks
s390: vfio-ap: implement mediated device open callback
s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
s390: vfio-ap: zeroize the AP queues.
s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
KVM: s390: CPU model support for AP virtualization
s390: doc: detailed specifications for AP virtualization
Documentation/s390/vfio-ap.txt | 616 +++++++++++++++++++++++
MAINTAINERS | 12 +
arch/s390/Kconfig | 11 +
arch/s390/include/asm/ap.h | 284 ++++++++++-
arch/s390/include/asm/kvm_host.h | 7 +
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/kvm-s390.c | 131 ++++--
arch/s390/kvm/kvm-s390.h | 1 +
arch/s390/kvm/vsie.c | 244 ++++++++-
arch/s390/tools/gen_facilities.c | 2 +
drivers/s390/crypto/Makefile | 4 +
drivers/s390/crypto/ap_asm.h | 236 ---------
drivers/s390/crypto/ap_bus.c | 21 +-
drivers/s390/crypto/ap_bus.h | 1 +
drivers/s390/crypto/ap_card.c | 1 -
drivers/s390/crypto/ap_queue.c | 1 -
drivers/s390/crypto/vfio_ap_drv.c | 137 +++++
drivers/s390/crypto/vfio_ap_ops.c | 885 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 106 ++++
drivers/s390/crypto/zcrypt_card.c | 12 +
drivers/s390/crypto/zcrypt_queue.c | 12 +
include/uapi/linux/vfio.h | 2 +
22 files changed, 2373 insertions(+), 354 deletions(-)
create mode 100644 Documentation/s390/vfio-ap.txt
delete mode 100644 drivers/s390/crypto/ap_asm.h
create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
create mode 100644 drivers/s390/crypto/vfio_ap_private.h
From: Tony Krowiak <[email protected]>
Introduces a new CPU model feature and two CPU model
facilities to support AP virtualization for KVM guests.
CPU model feature:
The KVM_S390_VM_CPU_FEAT_AP feature indicates that
AP instructions are available on the guest. This
feature will be enabled by the kernel only if the AP
instructions are installed on the linux host. This feature
must be specifically turned on for the KVM guest from
userspace to use the VFIO AP device driver for guest
access to AP devices.
CPU model facilities:
1. AP Query Configuration Information (QCI) facility is installed.
This is indicated by setting facilities bit 12 for
the guest. The kernel will not enable this facility
for the guest if it is not set on the host. This facility
must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
feature is not installed.
If this facility is not set for the KVM guest, then only
APQNs with an APQI less than 16 will be used by a Linux
guest regardless of the matrix configuration for the virtual
machine. This is a limitation of the Linux AP bus.
2. AP Facilities Test facility (APFT) is installed.
This is indicated by setting facilities bit 15 for
the guest. The kernel will not enable this facility for
the guest if it is not set on the host. This facility
must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
feature is not installed.
If this facility is not set for the KVM guest, then no
AP devices will be available to the guest regardless of
the guest's matrix configuration for the virtual
machine. This is a limitation of the Linux AP bus.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Christian Borntraeger <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
arch/s390/kvm/kvm-s390.c | 7 +++++++
arch/s390/tools/gen_facilities.c | 2 ++
2 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 9203f0b..7d4fe9b 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -367,6 +367,13 @@ static void kvm_s390_cpu_feat_init(void)
if (MACHINE_HAS_ESOP)
allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
+
+ /*
+ * Check if AP instructions installed on host
+ */
+ if (ap_instructions_available() == 0)
+ allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
+
/*
* We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
* 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
index 90a8c9e..a52290b 100644
--- a/arch/s390/tools/gen_facilities.c
+++ b/arch/s390/tools/gen_facilities.c
@@ -106,6 +106,8 @@ struct facility_def {
.name = "FACILITIES_KVM_CPUMODEL",
.bits = (int[]){
+ 12, /* AP Query Configuration Information */
+ 15, /* AP Facilities Test */
-1 /* END */
}
},
--
1.7.1
From: Tony Krowiak <[email protected]>
This patch provides documentation describing the AP architecture and
design concepts behind the virtualization of AP devices. It also
includes an example of how to configure AP devices for exclusive
use of KVM guests.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
Documentation/s390/vfio-ap.txt | 616 ++++++++++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
2 files changed, 617 insertions(+), 0 deletions(-)
create mode 100644 Documentation/s390/vfio-ap.txt
diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
new file mode 100644
index 0000000..c08c1c9
--- /dev/null
+++ b/Documentation/s390/vfio-ap.txt
@@ -0,0 +1,616 @@
+Introduction:
+============
+The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
+of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
+The AP devices provide cryptographic functions to all CPUs assigned to a
+linux system running in an IBM Z system LPAR.
+
+The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
+is to make AP cards available to KVM guests using the VFIO mediated device
+framework. This implementation relies considerably on the s390 virtualization
+facilities which do most of the hard work of providing direct access to AP
+devices.
+
+AP Architectural Overview:
+=========================
+To facilitate the comprehension of the design, let's start with some
+definitions:
+
+* AP adapter
+
+ An AP adapter is an IBM Z adapter card that can perform cryptographic
+ functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
+ assigned to the LPAR in which a linux host is running will be available to
+ the linux host. Each adapter is identified by a number from 0 to 255. When
+ installed, an AP adapter is accessed by AP instructions executed by any CPU.
+
+ The AP adapter cards are assigned to a given LPAR via the system's Activation
+ Profile which can be edited via the HMC. When the system is IPL'd, the AP bus
+ module is loaded and detects the AP adapter cards assigned to the LPAR. The AP
+ bus creates a sysfs device for each adapter as they are detected. For example,
+ if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will
+ create the following sysfs entries:
+
+ /sys/devices/ap/card04
+ /sys/devices/ap/card0a
+
+ Symbolic links to these devices will also be created in the AP bus devices
+ sub-directory:
+
+ /sys/bus/ap/devices/[card04]
+ /sys/bus/ap/devices/[card04]
+
+* AP domain
+
+ An adapter is partitioned into domains. Each domain can be thought of as
+ a set of hardware registers for processing AP instructions. An adapter can
+ hold up to 256 domains. Each domain is identified by a number from 0 to 255.
+ Domains can be further classified into two types:
+
+ * Usage domains are domains that can be accessed directly to process AP
+ commands.
+
+ * Control domains are domains that are accessed indirectly by AP
+ commands sent to a usage domain to control or change the domain; for
+ example, to set a secure private key for the domain.
+
+ The AP usage and control domains are assigned to a given LPAR via the system's
+ Activation Profile which can be edited via the HMC. When the system is IPL'd,
+ the AP bus module is loaded and detects the AP usage and control domains
+ assigned to the LPAR. The domain number of each usage domain will be coupled
+ with the adapter number of each AP adapter assigned to the LPAR to identify
+ the AP queues (see AP Queue section below). The domain number of each control
+ domain will be represented in a bitmask and stored in a sysfs file
+ /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the mask,
+ from most to least significant bit, correspond to domains 0-255.
+
+ A domain may be assigned to a system as both a usage and control domain, or
+ as a control domain only. Consequently, all domains assigned as both a usage
+ and control domain can both process AP commands as well as be changed by an AP
+ command sent to any usage domain assigned to the same system. Domains assigned
+ only as control domains can not process AP commands but can be changed by AP
+ commands sent to any usage domain assigned to the system.
+
+* AP Queue
+
+ An AP queue is the means by which an AP command-request message is sent to a
+ usage domain inside a specific adapter. An AP queue is identified by a tuple
+ comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
+ APQI corresponds to a given usage domain number within the adapter. This tuple
+ forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
+ instructions include a field containing the APQN to identify the AP queue to
+ which the AP command-request message is to be sent for processing.
+
+ The AP bus will create a sysfs device for each APQN that can be derived from
+ the cross product of the AP adapter and usage domain numbers detected when the
+ AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
+ domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
+ following sysfs entries:
+
+ /sys/devices/ap/card04/04.0006
+ /sys/devices/ap/card04/04.0047
+ /sys/devices/ap/card0a/0a.0006
+ /sys/devices/ap/card0a/0a.0047
+
+ The following symbolic links to these devices will be created in the AP bus
+ devices subdirectory:
+
+ /sys/bus/ap/devices/[04.0006]
+ /sys/bus/ap/devices/[04.0047]
+ /sys/bus/ap/devices/[0a.0006]
+ /sys/bus/ap/devices/[0a.0047]
+
+* AP Instructions:
+
+ There are three AP instructions:
+
+ * NQAP: to enqueue an AP command-request message to a queue
+ * DQAP: to dequeue an AP command-reply message from a queue
+ * PQAP: to administer the queues
+
+AP and SIE:
+==========
+Let's now take a look at how AP instructions executed on a guest are interpreted
+by the hardware.
+
+A satellite control block called the Crypto Control Block (CRYCB) is attached to
+our main hardware virtualization control block. The CRYCB contains three fields
+to identify the adapters, usage domains and control domains assigned to the KVM
+guest:
+
+* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
+ to the KVM guest. Each bit in the mask, from most significant to least
+ significant bit, corresponds to an APID from 0-255. If a bit is set, the
+ corresponding adapter is valid for use by the KVM guest.
+
+* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
+ assigned to the KVM guest. Each bit in the mask, from most significant to
+ least significant bit, corresponds to an AP queue index (APQI) from 0-255. If
+ a bit is set, the corresponding queue is valid for use by the KVM guest.
+
+* The AP Domain Mask field is a bit mask that identifies the AP control domains
+ assigned to the KVM guest. The ADM bit mask controls which domains can be
+ changed by an AP command-request message sent to a usage domain from the
+ guest. Each bit in the mask, from least significant to most significant bit,
+ corresponds to a domain from 0-255. If a bit is set, the corresponding domain
+ can be modified by an AP command-request message sent to a usage domain
+ configured for the KVM guest.
+
+If you recall from the description of an AP Queue, AP instructions include
+an APQN to identify the AP adapter and AP queue to which an AP command-request
+message is to be sent (NQAP and PQAP instructions), or from which a
+command-reply message is to be received (DQAP instruction). The validity of an
+APQN is defined by the matrix calculated from the APM and AQM; it is the
+cross product of all assigned adapter numbers (APM) with all assigned queue
+indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
+assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
+the guest.
+
+The APQNs can provide secure key functionality - i.e., a private key is stored
+on the adapter card for each of its domains - so each APQN must be assigned to
+at most one guest or to the linux host.
+
+ Example 1: Valid configuration:
+ ------------------------------
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapter 1,2 domain 7
+
+ This is valid because both guests have a unique set of APQNs: Guest1 has
+ APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7).
+
+ Example 2: Invalid configuration:
+ Guest1: adapters 1,2 domains 5,6
+ Guest2: adapter 1 domains 6,7
+
+ This is an invalid configuration because both guests have access to
+ APQN (1,6).
+
+The Design:
+===========
+The design introduces three new objects:
+
+1. AP matrix device
+2. VFIO AP device driver (vfio_ap.ko)
+3. AP mediated matrix passthrough device
+
+The VFIO AP device driver
+-------------------------
+The VFIO AP (vfio_ap) device driver serves the following purposes:
+
+1. Provides the interfaces to bind APQNs for exclusive use of KVM guests.
+
+2. Sets up the VFIO mediated device interfaces to manage a mediated matrix
+ device and creates the sysfs interfaces for assigning adapters, usage
+ domains, and control domains comprising the matrix for a KVM guest.
+
+3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's
+ SIE state description to grant the guest access to a matrix of AP devices
+
+Reserve APQNs for exclusive use of KVM guests
+---------------------------------------------
+The following block diagram illustrates the mechanism by which APQNs are
+reserved:
+
+ +------------------+
+ remove | |
+ +------------------->+ cex4queue driver +
+ | | |
+ | +------------------+
+ |
+ |
+ | remove +------------------+
+ | +-----------------+ |<---------------+
+ | | probe | Device core | |
+ | | +--------------+ +<-----------+ |
+ | | | +--------+---------+ | |
+ | | | ^ | |
+ | | | register | | |
+ | | | vfio_ap device | bind | | unbind
+ | v v | vfio_ap | | cex4queue
++--------+-----+---+ +--------+---------+ +-+---+---+--+
+| | register | | | |
+| ap_bus +<---------+ vfio_ap driver + + admin |
+| +--------->+ | | |
++------------------+ probe +---+--------+-----+ +------------+
+ | |
+ create | | assign
+ | | adapters/domains/control domains
+ v v
+ +---+--------+-----+
+ | |
+ | mediated device |
+ | |
+ +------------------+
+
+The process for reserving an AP queue for use by a KVM guest is:
+
+* The vfio-ap driver during its initialization will perform the following:
+ * Create a single 'vfio_ap' device, /sys/devices/virtual/misc/vfio_ap,
+ otherwise known as the matrix device.
+ * Register the matrix device with the device core
+* Register with the ap_bus for AP queue devices of type 10 (CEX4 and
+ newer) and to provide the vfio_ap driver's probe and remove callback
+ interfaces. Devices older than CEX4 queues are not supported to simplify the
+ implementation and because older devices will be going out of service in the
+ relatively near future.
+* The admin needs to unbind AP Queues to be reserved for use by guests from
+ the cex4queue device driver and bind them to the vfio_ap device driver.
+
+
+Set up the VFIO mediated device interfaces
+------------------------------------------
+The VFIO AP device driver utilizes the common interface of the VFIO mediated
+device core driver to:
+* Register an AP mediated bus driver to add a mediated matrix device to and
+ remove it from a VFIO group.
+* Create and destroy a mediated matrix device
+* Add a mediated matrix device to and remove it from the AP mediated bus driver
+* Add a mediated matrix device to and remove it from an IOMMU group
+
+The following high-level block diagram shows the main components and interfaces
+of the VFIO AP mediated matrix device driver:
+
+ +-------------+
+ | |
+ | +---------+ | mdev_register_driver() +--------------+
+ | | Mdev | +<-----------------------+ |
+ | | bus | | | vfio_mdev.ko |
+ | | driver | +----------------------->+ |<-> VFIO user
+ | +---------+ | probe()/remove() +--------------+ APIs
+ | |
+ | MDEV CORE |
+ | MODULE |
+ | mdev.ko |
+ | +---------+ | mdev_register_device() +--------------+
+ | |Physical | +<-----------------------+ |
+ | | device | | | vfio_ap.ko |<-> matrix
+ | |interface| +----------------------->+ | device
+ | +---------+ | callback +--------------+
+ +-------------+
+
+During initialization of the vfio_ap module, the matrix device is registered
+with an 'mdev_parent_ops' structure that provides the sysfs attribute
+structures, mdev functions and callback interfaces for managing the mediated
+matrix device.
+
+* sysfs attribute structures:
+ * supported_type_groups
+ The VFIO mediated device framework supports creation of user-defined
+ mediated device types. These mediated device types are specified
+ via the 'supported_type_groups' structure when a device is registered
+ with the mediated device framework. The registration process creates the
+ sysfs structures for each mediated device type specified in the
+ 'mdev_supported_types' sub-directory of the device being registered. Along
+ with the device type, the sysfs attributes of the mediated device type are
+ provided.
+
+ The VFIO AP device driver will register one mediated device type for
+ passthrough devices:
+ /sys/devices/virtual/misc/vfio_ap/mdev_supported_types/vfio_ap-passthrough
+ Only the read-only attributes required by the VFIO mdev framework will
+ be provided:
+ ... name
+ ... device_api
+ ... available_instances
+ ... device_api
+ Where:
+ * name: specifies the name of the mediated device type
+ * device_api: the mediated device type's API
+ * available_instances: the number of mediated matrix passthrough devices
+ that can be created
+ * device_api: specifies the VFIO API
+ * mdev_attr_groups
+ This attribute group identifies the user-defined sysfs attributes of the
+ mediated device. When a device is registered with the VFIO mediated device
+ framework, the sysfs attributes files identified in the 'mdev_attr_groups'
+ structure will be created in the mediated matrix device's directory. The
+ sysfs attributes for a mediated matrix device are:
+ * assign_adapter:
+ * unassign_adapter:
+ Write-only attributes for assigning/unassigning an AP adapter to/from the
+ mediated matrix device. To assign/unassign an adapter, the APID of the
+ adapter is written to the respective attribute file.
+ * assign_domain:
+ * unassign_domain:
+ Write-only attributes for assigning/unassigning an AP usage domain to/from
+ the mediated matrix device. To assign/unassign a domain, the APQI of the
+ AP queue corresponding to a usage domain is written to the respective
+ attribute file.
+ * matrix:
+ A read-only file for displaying the APQNs derived from the cross product
+ of the adapters and domains assigned to the mediated matrix device.
+ * assign_control_domain:
+ * unassign_control_domain:
+ Write-only attributes for assigning/unassigning an AP control domain
+ to/from the mediated matrix device. To assign/unassign a control domain,
+ the ID of a domain to be assigned/unassigned is written to the respective
+ attribute file.
+ * control_domains:
+ A read-only file for displaying the control domain numbers assigned to the
+ mediated matrix device.
+
+* functions:
+ * create:
+ allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
+ * Store the reference to the KVM structure for the guest using the mdev
+ * Store the AP matrix configuration for the adapters, domains, and control
+ domains assigned via the corresponding sysfs attributes files
+ * remove:
+ deallocates the mediated matrix device's ap_matrix_mdev structure. This will
+ be allowed only if a running guest is not using the mdev.
+
+* callback interfaces
+ * open:
+ The vfio_ap driver uses this callback to register a
+ VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix
+ device. The open is invoked when QEMU connects the VFIO iommu group
+ for the mdev matrix device to the MDEV bus. Access to the KVM structure used
+ to configure the KVM guest is provided via this callback. The KVM structure,
+ is used to configure the guest's access to the AP matrix defined via the
+ mediated matrix device's sysfs attribute files.
+ * release:
+ unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
+ mdev matrix device and deconfigures the guest's AP matrix.
+
+Configure the APM, AQM and ADM in the CRYCB:
+-------------------------------------------
+Configuring the AP matrix for a KVM guest will be performed when the
+VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
+function is called when QEMU connects to KVM. The CRYCB is configured by:
+* Setting the bits in the APM corresponding to the APIDs assigned to the
+ mediated matrix device via its 'assign_adapter' interface.
+* Setting the bits in the AQM corresponding to the APQIs assigned to the
+ mediated matrix device via its 'assign_domain' interface.
+* Setting the bits in the ADM corresponding to the domain dIDs assigned to the
+ mediated matrix device via its 'assign_control_domains' interface.
+
+The CPU model features for AP
+-----------------------------
+The AP stack relies on the presence of the AP instructions as well as two
+facilities: The AP Facilities Test (APFT) facility; and the AP Query
+Configuration Information (QCI) facility. These features/facilities are made
+available to a KVM guest via the following CPU model features:
+
+1. ap: Indicates whether the AP instructions are installed on the guest. This
+ feature will be enabled by KVM only if the AP instructions are installed
+ on the host.
+
+2. apft: Indicates the APFT facility is available on the guest. This facility
+ can be made available to the guest only if it is available on the host.
+
+3. apft: Indicates the AP QCI facility is available on the guest. This facility
+ can be made available to the guest only if it is available on the host.
+
+Note that if the user chooses to specify a CPU model different than the 'host'
+model to QEMU, the CPU model features and facilities need to be turned on
+explicitly; for example:
+
+ /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on
+
+A guest can be precluded from using AP features/facilities by turning them off
+explicitly; for example:
+
+ /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off
+
+Example:
+=======
+Let's now provide an example to illustrate how KVM guests may be given
+access to AP facilities. For this example, we will show how to configure
+two guests such that executing the lszcrypt command on the guests would
+look like this:
+
+Guest1
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5C CCA-Coproc
+05.0004 CEX5C CCA-Coproc
+05.00ab CEX5C CCA-Coproc
+06 CEX5A Accelerator
+06.0004 CEX5A Accelerator
+06.00ab CEX5C CCA-Coproc
+
+Guest2
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5A Accelerator
+05.0047 CEX5A Accelerator
+05.00ff CEX5A Accelerator
+
+These are the steps:
+
+1. Install the vfio_ap module on the linux host. The dependency chain for the
+ vfio_ap module is:
+ * vfio
+ * mdev
+ * vfio_mdev
+ * KVM
+ * vfio_ap
+
+2. Secure the AP queues to be used by the two guests so that the host can not
+ access them. Only type 10 adapters (i.e., CEX4 and later) are supported
+ for the following reasons: To simplify the implementation; a lack of older
+ systems on which to test; and because the older hardware will go out of
+ service in a relatively short time.
+
+ To secure the AP queues each, each AP Queue device must first be unbound from
+ the cex4queue device driver. The sysfs location of the driver is:
+
+ /sys/bus/ap
+ --- [drivers]
+ ------ [cex4queue]
+ --------- [05.0004]
+ --------- [05.0047]
+ --------- [05.00ab]
+ --------- [05.00ff]
+ --------- [06.0004]
+ --------- [06.00ab]
+ --------- unbind
+
+ To unbind AP queue 05.0004 for example;
+
+ echo 05.0004 > unbind
+
+ The AP queue devices must then be bound to the vfio_ap driver. The sysfs
+ location of the driver is:
+
+ /sys/bus/ap
+ --- [drivers]
+ ------ [cex4queue]
+ ---------- bind
+
+ To bind AP queue 05.0004 to the vfio_ap driver:
+
+ echo 05.0004 > bind
+
+ Take note that the AP queues bound to the vfio_ap driver will be available
+ for guest usage until the vfio_ap module is unloaded, or the host system is
+ shut down.
+
+3. Create the mediated devices needed to configure the AP matrixes for the
+ two guests and to provide an interface to the vfio_ap driver for
+ use by the guests:
+
+ /sys/devices/virtual/misc
+ --- [vfio_ap]
+ --------- [mdev_supported_types]
+ ------------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
+ --------------- create
+ --------------- [devices]
+
+ To create the mediated devices for the two guests:
+
+ uuidgen > create
+ uuidgen > create
+
+ This will create two mediated devices in the [devices] subdirectory named
+ with the UUID written to the create attribute file. We call them $uuid1
+ and $uuid2:
+
+ /sys/devices/virtual/misc/
+ --- [vfio_ap]
+ --------- [mdev_supported_types]
+ ------------ [vfio_ap-passthrough]
+ --------------- [devices]
+ ------------------ [$uuid1]
+ --------------------- assign_adapter
+ --------------------- assign_control_domain
+ --------------------- assign_domain
+ --------------------- matrix
+ --------------------- unassign_adapter
+ --------------------- unassign_control_domain
+ --------------------- unassign_domain
+
+ ------------------ [$uuid2]
+ --------------------- assign_adapter
+ --------------------- assign_control_domain
+ --------------------- assign_domain
+ --------------------- matrix
+ --------------------- unassign_adapter
+ --------------------- unassign_control_domain
+ --------------------- unassign_domain
+
+4. The administrator now needs to configure the matrixes for mediated
+ devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
+
+ This is how the matrix is configured for Guest1:
+
+ echo 5 > assign_adapter
+ echo 6 > assign_adapter
+ echo 4 > assign_domain
+ echo 0xab > assign_domain
+
+ For this implementation, all usage domains - i.e., domains assigned
+ via the assign_domain attribute file - will also be configured in the ADM
+ field of the KVM guest's CRYCB, so there is no need to assign control
+ domains here unless you want to assign control domains that are not
+ assigned as usage domains.
+
+ If a mistake is made configuring an adapter, domain or control domain,
+ you can use the unassign_xxx files to unassign the adapter, domain or
+ control domain.
+
+ To display the matrix configuration for Guest1:
+
+ cat matrix
+
+ This is how the matrix is configured for Guest2:
+
+ echo 5 > assign_adapter
+ echo 0x47 > assign_domain
+ echo 0xff > assign_domain
+
+ In order to successfully assign an adapter:
+
+ * All APQNs that can be derived from the adapter ID and the IDs of
+ the previously assigned domains must be bound to the vfio_ap device
+ driver. If no domains have yet been assigned, then there must be at least
+ one APQN with the specified APID bound to the vfio_ap driver.
+
+ No APQN that can be derived from the adapter ID and the IDs of the
+ previously assigned domains can be assigned to another mediated matrix
+ device.
+
+ In order to successfully assign a domain:
+
+ * All APQNs that can be derived from the domain ID and the IDs of
+ the previously assigned adapters must be bound to the vfio_ap device
+ driver. If no domains have yet been assigned, then there must be at least
+ one APQN with the specified APQI bound to the vfio_ap driver.
+
+ No APQN that can be derived from the domain ID and the IDs of the
+ previously assigned adapters can be assigned to another mediated matrix
+ device.
+
+5. Start Guest1:
+
+ /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on,apqci=on,apft=on \
+ -device vfio-ap,sysfsdev=/sys/devices/virtual/misc/vfio_ap/$uuid1 ...
+
+7. Start Guest2:
+
+ /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on,apqci=on,apft=on \
+ -device vfio-ap,sysfsdev=/sys/devices/virtual/misc/vfio_ap/$uuid2 ...
+
+When the guest is shut down, the mediated matrix device may be removed.
+
+Using our example again, to remove the mediated matrix device $uuid1:
+
+ /sys/devices/virtual/misc
+ --- [vfio_ap]
+ --------- [mdev_supported_types]
+ ------------ [vfio_ap-passthrough]
+ --------------- [devices]
+ ------------------ [$uuid1]
+ --------------------- remove
+
+
+ echo 1 > remove
+
+ This will release all the AP queues configured for the mediated device and
+ remove all of the mdev matrix device's sysfs structures including the mdev
+ device itself. To recreate and reconfigure the mdev matrix device, all of the
+ steps starting with step 3 will have to be performed again. Note that the
+ remove will fail if a guest using the mdev is still running.
+
+ It is not necessary to remove an mdev matrix device, but one may want to
+ remove it if no guest will use it during the lifetime of the linux host. If
+ the mdev matrix device is removed, one may want to unbind the AP queues the
+ guest was using from the vfio_ap device driver and bind them back to the
+ default driver. Alternatively, the AP queues can be configured for another
+ mdev matrix (i.e., guest).
+
+
+Limitations
+===========
+* The KVM/kernel interfaces do not provide a way to prevent unbinding an AP
+ queue that is still assigned to a mediated device. Even if the device
+ 'remove' callback returns an error, the device core detaches the AP
+ queue from the VFIO AP driver. It is therefore incumbent upon the
+ administrator to make sure there is no mediated device to which the
+ APQN - for the AP queue being unbound - is assigned.
+
+* Hot plug/unplug of AP devices is not supported for guests.
+
+* Live guest migration is not supported for guests using AP devices.
\ No newline at end of file
diff --git a/MAINTAINERS b/MAINTAINERS
index 3597910..dc25477 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12428,6 +12428,7 @@ S: Supported
F: drivers/s390/crypto/vfio_ap_drv.c
F: drivers/s390/crypto/vfio_ap_private.h
F: drivers/s390/crypto/vfio_ap_ops.c
+F: Documentation/s390/vfio-ap.txt
S390 ZFCP DRIVER
M: Steffen Maier <[email protected]>
--
1.7.1
From: Tony Krowiak <[email protected]>
Implements the VFIO_DEVICE_RESET ioctl. This ioctl zeroizes
all of the AP queues assigned to the guest.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Reviewed-by: Pierre Morel <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index e5c6dc1..d25193a 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -835,7 +835,7 @@ static int vfio_ap_mdev_get_device_info(unsigned long arg)
if (info.argsz < minsz)
return -EINVAL;
- info.flags = VFIO_DEVICE_FLAGS_AP;
+ info.flags = VFIO_DEVICE_FLAGS_AP | VFIO_DEVICE_FLAGS_RESET;
info.num_regions = 0;
info.num_irqs = 0;
@@ -851,6 +851,9 @@ static ssize_t vfio_ap_mdev_ioctl(struct mdev_device *mdev,
case VFIO_DEVICE_GET_INFO:
ret = vfio_ap_mdev_get_device_info(arg);
break;
+ case VFIO_DEVICE_RESET:
+ ret = vfio_ap_mdev_reset_queues(mdev, true);
+ break;
default:
ret = -EOPNOTSUPP;
break;
--
1.7.1
From: Pierre Morel <[email protected]>
When we clear the Crypto Control Block (CRYCB) used by a guest
level 2, the vSIE shadow CRYCB for guest level 3 must be updated
before the guest uses it.
We achieve this by using the KVM_REQ_VSIE_RESTART synchronous
request for each vCPU belonging to the guest to force the reload
of the shadow CRYCB before rerunning the guest level 3.
Signed-off-by: Pierre Morel <[email protected]>
---
arch/s390/kvm/kvm-s390.c | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 0d03249..9203f0b 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -1921,6 +1921,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
void kvm_arch_crypto_clear_masks(struct kvm *kvm)
{
+ int i;
+ struct kvm_vcpu *vcpu;
+
mutex_lock(&kvm->lock);
kvm_s390_vcpu_block_all(kvm);
@@ -1929,6 +1932,10 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm)
memset(&kvm->arch.crypto.crycb->apcb1, 0,
sizeof(kvm->arch.crypto.crycb->apcb1));
+ /* recreate the shadow crycb for each vcpu */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
+
kvm_s390_vcpu_unblock_all(kvm);
mutex_unlock(&kvm->lock);
}
--
1.7.1
From: Tony Krowiak <[email protected]>
Let's call PAPQ(ZAPQ) to zeroize a queue:
* For each queue configured for a mediated matrix device
when it is released.
Zeroizing a queue resets the queue, clears all pending
messages for the queue entries and disables adapter interruptions
associated with the queue.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 29 ++++++++++++++++++++++++++++-
drivers/s390/crypto/vfio_ap_private.h | 25 +++++++++++++++++++++++++
2 files changed, 53 insertions(+), 1 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 31e0dbe..e5c6dc1 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -761,6 +761,30 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
return NOTIFY_OK;
}
+static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev, bool force)
+{
+ int ret;
+ int rc = 0;
+ unsigned long apid, apqi;
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
+ matrix_mdev->matrix.apm_max + 1) {
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+ matrix_mdev->matrix.aqm_max + 1) {
+ ret = vfio_ap_reset_queue(apid, apqi, 1);
+ if (ret) {
+ if (force)
+ rc = ret;
+ else
+ return ret;
+ }
+ }
+ }
+
+ return rc;
+}
+
static int vfio_ap_mdev_open(struct mdev_device *mdev)
{
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
@@ -788,7 +812,10 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
{
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
- kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+ if (matrix_mdev->kvm)
+ kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+
+ vfio_ap_mdev_reset_queues(mdev, true);
vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
&matrix_mdev->group_notifier);
matrix_mdev->kvm = NULL;
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 6fa77cf..087e21a 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -78,4 +78,29 @@ struct ap_matrix_mdev {
extern int vfio_ap_mdev_register(void);
extern void vfio_ap_mdev_unregister(void);
+static inline int vfio_ap_reset_queue(unsigned int apid, unsigned int apqi,
+ unsigned int retry)
+{
+ struct ap_queue_status status;
+
+ do {
+ status = ap_zapq(AP_MKQID(apid, apqi));
+ switch (status.response_code) {
+ case AP_RESPONSE_NORMAL:
+ return 0;
+ case AP_RESPONSE_RESET_IN_PROGRESS:
+ case AP_RESPONSE_BUSY:
+ msleep(20);
+ break;
+ default:
+ pr_warn("%s: error zeroizing %02x.%04x: response code %d\n",
+ VFIO_AP_MODULE_NAME, apid, apqi,
+ status.response_code);
+ return -EIO;
+ }
+ } while (retry--);
+
+ return -EBUSY;
+}
+
#endif /* _VFIO_AP_PRIVATE_H_ */
--
1.7.1
From: Tony Krowiak <[email protected]>
Introduces ioctl access to the VFIO AP Matrix device driver
by implementing the VFIO_DEVICE_GET_INFO ioctl. This ioctl
provides the VFIO AP Matrix device driver information to the
guest machine.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Pierre Morel <[email protected]>
Acked-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 38 +++++++++++++++++++++++++++++++++++++
include/uapi/linux/vfio.h | 1 +
2 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index ea96b95..31e0dbe 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -795,6 +795,43 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
module_put(THIS_MODULE);
}
+static int vfio_ap_mdev_get_device_info(unsigned long arg)
+{
+ unsigned long minsz;
+ struct vfio_device_info info;
+
+ minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz)
+ return -EINVAL;
+
+ info.flags = VFIO_DEVICE_FLAGS_AP;
+ info.num_regions = 0;
+ info.num_irqs = 0;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+}
+
+static ssize_t vfio_ap_mdev_ioctl(struct mdev_device *mdev,
+ unsigned int cmd, unsigned long arg)
+{
+ int ret;
+
+ switch (cmd) {
+ case VFIO_DEVICE_GET_INFO:
+ ret = vfio_ap_mdev_get_device_info(arg);
+ break;
+ default:
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
@@ -803,6 +840,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
.remove = vfio_ap_mdev_remove,
.open = vfio_ap_mdev_open,
.release = vfio_ap_mdev_release,
+ .ioctl = vfio_ap_mdev_ioctl,
};
int vfio_ap_mdev_register(void)
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index bfbe2be..f378b98 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -200,6 +200,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */
#define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
#define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
+#define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */
};
--
1.7.1
From: Pierre Morel <[email protected]>
Shadowing the crypto control block now supports APCB shadowing.
AP instruction interpretation for guest 3 through ECA.28 is shadowed when
guest 2 ECA.28 is set.
CRYCB is shadowed for APCB and wrapping keys.
CRYCB format 0 is now supported for both guests 2 and 3.
Shadow CRYCB always uses the guest 2 CRYCB format and it
follows that:
* Guest 3 CRYCB format 0 is supported with guest 2 CRYCB format 0,1 or 2
* Guest 3 CRYCB format 1 is supported with guest 2 CRYCB format 1 or 2
* Guest 3 CRYCB format 2 is supported with guest 2 CRYCB format 2
Signed-off-by: Pierre Morel <[email protected]>
Reviewed-by: Tony Krowiak <[email protected]>
Acked-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
arch/s390/kvm/vsie.c | 224 ++++++++++++++++++++++++++++++++++++++++++++------
1 files changed, 200 insertions(+), 24 deletions(-)
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index aa30b48..fc7986b 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -136,17 +136,8 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return 0;
}
-/*
- * Create a shadow copy of the crycb block and setup key wrapping, if
- * requested for guest 3 and enabled for guest 2.
- *
- * We only accept format-1 (no AP in g2), but convert it into format-2
- * There is nothing to do for format-0.
- *
- * Returns: - 0 if shadowed or nothing to do
- * - > 0 if control has to be given to guest 2
- */
-static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+/* Copy keys into shadow crycb, is only called if MSA3 is available. */
+static int copy_key_masks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
{
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
@@ -155,30 +146,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
unsigned long *b1, *b2;
u8 ecb3_flags;
- scb_s->crycbd = 0;
- if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))
- return 0;
- /* format-1 is supported with message-security-assist extension 3 */
- if (!test_kvm_facility(vcpu->kvm, 76))
- return 0;
/* we may only allow it if enabled for guest 2 */
ecb3_flags = scb_o->ecb3 & vcpu->arch.sie_block->ecb3 &
(ECB3_AES | ECB3_DEA);
if (!ecb3_flags)
return 0;
- if ((crycb_addr & PAGE_MASK) != ((crycb_addr + 128) & PAGE_MASK))
- return set_validity_icpt(scb_s, 0x003CU);
- else if (!crycb_addr)
- return set_validity_icpt(scb_s, 0x0039U);
-
/* copy only the wrapping keys */
if (read_guest_real(vcpu, crycb_addr + 72, &vsie_page->crycb, 56))
return set_validity_icpt(scb_s, 0x0035U);
scb_s->ecb3 |= ecb3_flags;
- scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | CRYCB_FORMAT1 |
- CRYCB_FORMAT2;
/* xor both blocks in one run */
b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
@@ -189,6 +167,204 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return 0;
}
+/* Copy masks into apcb when g2 and g3 use format 1 */
+static int copy_apcb1(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+ struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
+ struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+ const uint32_t crycbd_o = READ_ONCE(scb_o->crycbd);
+ const u32 crycb_o = crycbd_o & 0x7ffffff8U;
+ struct kvm_s390_crypto_cb *crycb_h = &vcpu->kvm->arch.sie_page2->crycb;
+ struct kvm_s390_crypto_cb *crycb_s = &vsie_page->crycb;
+ unsigned long *apcb_s = (unsigned long *) &crycb_s->apcb1;
+ unsigned long *apcb_h = (unsigned long *) &crycb_h->apcb1;
+ int i;
+ u32 src;
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb1);
+ if (read_guest_real(vcpu, src, apcb_s, sizeof(struct kvm_s390_apcb1)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ for (i = 0; i < sizeof(struct kvm_s390_apcb1); i += sizeof(*apcb_s))
+ *apcb_s &= *apcb_h;
+
+ return 0;
+}
+
+/*
+ * Copy masks into apcb when g2 use format 1 and g3 use format 0
+ * In this case the shadow APCB uses format 1
+ */
+static int copy_apcb01(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+ struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
+ struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+ const uint32_t crycbd_o = READ_ONCE(scb_o->crycbd);
+ const u32 crycb_o = crycbd_o & 0x7ffffff8U;
+ struct kvm_s390_apcb1 *apcb_h = &vcpu->kvm->arch.sie_page2->crycb.apcb1;
+ struct kvm_s390_apcb1 *apcb_s = &vsie_page->crycb.apcb1;
+ u32 src;
+
+ memset(apcb_s, 0, sizeof(*apcb_s));
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb0.apm[0]);
+ if (read_guest_real(vcpu, src, &apcb_s->apm[0], sizeof(__u64)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb0.aqm[0]);
+ if (read_guest_real(vcpu, src, &apcb_s->aqm[0], sizeof(__u64)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb0.adm[0]);
+ if (read_guest_real(vcpu, src, &apcb_s->adm[0], sizeof(__u64)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ apcb_s->apm[0] &= apcb_h->apm[0];
+ apcb_s->aqm[0] &= apcb_h->aqm[0];
+ apcb_s->adm[0] &= apcb_h->adm[0];
+
+ return 0;
+}
+
+/* Copy masks into apcb when g2 and g3 use format 0 */
+static int copy_apcb0(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+ struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
+ struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+ const uint32_t crycbd_o = READ_ONCE(scb_o->crycbd);
+ const u32 crycb_o = crycbd_o & 0x7ffffff8U;
+ struct kvm_s390_apcb0 *apcb_h = &vcpu->kvm->arch.sie_page2->crycb.apcb0;
+ struct kvm_s390_apcb0 *apcb_s = &vsie_page->crycb.apcb0;
+ u32 src;
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb0.apm[0]);
+ if (read_guest_real(vcpu, src, &apcb_s->apm[0], sizeof(__u64)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb0.aqm[0]);
+ if (read_guest_real(vcpu, src, &apcb_s->aqm[0], sizeof(__u64)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ src = crycb_o + offsetof(struct kvm_s390_crypto_cb, apcb0.adm[0]);
+ if (read_guest_real(vcpu, src, &apcb_s->adm[0], sizeof(__u64)))
+ return set_validity_icpt(scb_s, 0x0035U);
+
+ apcb_s->apm[0] &= apcb_h->apm[0];
+ apcb_s->aqm[0] &= apcb_h->aqm[0];
+ apcb_s->adm[0] &= apcb_h->adm[0];
+
+ return 0;
+}
+
+/* Shadowing APCB depends on G2 and G3 CRYCB format */
+static int copy_apcb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page,
+ int g2_fmt, int g3_fmt)
+{
+ struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+ int ret = 0;
+
+ switch (g2_fmt) {
+ case CRYCB_FORMAT0:
+ switch (g3_fmt) {
+ case CRYCB_FORMAT0:
+ ret = copy_apcb0(vcpu, vsie_page);
+ break;
+ default:
+ return set_validity_icpt(scb_s, 0x0020U);
+ }
+ break;
+ case CRYCB_FORMAT1:
+ switch (g3_fmt) {
+ case CRYCB_FORMAT1:
+ case CRYCB_FORMAT0: /* Fall through to copy APCB */
+ ret = copy_apcb0(vcpu, vsie_page);
+ break;
+ default:
+ return set_validity_icpt(scb_s, 0x0020U);
+ }
+ break;
+ case CRYCB_FORMAT2:
+ switch (g3_fmt) {
+ case CRYCB_FORMAT0:
+ case CRYCB_FORMAT1:
+ ret = copy_apcb01(vcpu, vsie_page);
+ break;
+ case CRYCB_FORMAT2:
+ ret = copy_apcb1(vcpu, vsie_page);
+ break;
+ }
+ break;
+ default:
+ /*
+ * Guest 2 format is valid or we can not get to here.
+ */
+ break;
+ }
+
+ return ret;
+}
+
+/*
+ * Create a shadow copy of the crycb block.
+ * - Setup key wrapping, if requested for guest 3 and enabled for guest 2.
+ * - Shadow APCB if requested by guest 3 and enabled for guest 2 through
+ * ECA_APIE.
+ *
+ * We only accept format-1 (no AP in g2), but convert it into format-2
+ * There is nothing to do for format-0.
+ *
+ * Returns: - 0 if shadowed or nothing to do
+ * - > 0 if control has to be given to guest 2
+ * - < 0 if something went wrong on copy
+ */
+#define ECA_APIE 0x00000008
+static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+ struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+ struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
+ const uint32_t crycbd_o = READ_ONCE(scb_o->crycbd);
+ const u32 crycb_addr = crycbd_o & 0x7ffffff8U;
+ int g2_fmt = vcpu->arch.sie_block->crycbd & CRYCB_FORMAT_MASK;
+ int g3_fmt = crycbd_o & CRYCB_FORMAT_MASK;
+ int g2_apie, g2_msa3, g3_apie, g3_msa3;
+ int size, ret;
+
+ /* crycb should not cross a page boundary */
+ size = (g3_fmt == CRYCB_FORMAT2) ? 0x100 : 0x80;
+ if ((crycb_addr & PAGE_MASK) != ((crycb_addr + size) & PAGE_MASK))
+ return set_validity_icpt(scb_s, 0x003CU);
+
+ g2_apie = vcpu->arch.sie_block->eca & ECA_APIE;
+ g3_apie = scb_o->eca & g2_apie;
+
+ g2_msa3 = test_kvm_facility(vcpu->kvm, 76);
+ g3_msa3 = (g3_fmt != CRYCB_FORMAT0) & g2_msa3;
+
+ scb_s->crycbd = 0;
+ /* If no AP instructions and no keys we just set crycbd to 0 */
+ if (!(g3_apie || g3_msa3))
+ return 0;
+
+ if (!crycb_addr)
+ return set_validity_icpt(scb_s, 0x0039U);
+
+ if (g3_apie) {
+ ret = copy_apcb(vcpu, vsie_page, g2_fmt, g3_fmt);
+ if (ret)
+ goto out;
+ scb_s->eca |= g3_apie;
+ }
+
+ if (g3_msa3)
+ ret = copy_key_masks(vcpu, vsie_page);
+
+ if (!ret)
+ scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | g2_fmt;
+
+out:
+ return ret;
+}
+
/* shadow (round up/down) the ibc to avoid validity icpt */
static void prepare_ibc(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
{
--
1.7.1
From: Tony Krowiak <[email protected]>
Implements the open callback on the mediated matrix device.
The function registers a group notifier to receive notification
of the VFIO_GROUP_NOTIFY_SET_KVM event. When notified,
the vfio_ap device driver will get access to the guest's
kvm structure. The open callback must ensure that only one
mediated device shall be opened per guest.
Signed-off-by: Tony Krowiak <[email protected]>
Acked-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Acked-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 175 ++++++++++++++++++++++++++++++++-
drivers/s390/crypto/vfio_ap_private.h | 2 +
2 files changed, 176 insertions(+), 1 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 61a25d4..ea96b95 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -13,6 +13,10 @@
#include <linux/device.h>
#include <linux/list.h>
#include <linux/ctype.h>
+#include <linux/bitops.h>
+#include <linux/kvm_host.h>
+#include <linux/module.h>
+#include <asm/kvm.h>
#include "vfio_ap_private.h"
@@ -55,6 +59,9 @@ static int vfio_ap_mdev_remove(struct mdev_device *mdev)
{
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
mutex_lock(&matrix_dev.lock);
list_del(&matrix_mdev->list);
mutex_unlock(&matrix_dev.lock);
@@ -291,6 +298,10 @@ static ssize_t assign_adapter_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_apid = matrix_mdev->matrix.apm_max;
+ /* If the guest is running, disallow assignment of adapter */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -348,6 +359,10 @@ static ssize_t unassign_adapter_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ /* If the guest is running, disallow un-assignment of adapter */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apid);
if (ret)
return ret;
@@ -393,6 +408,10 @@ static ssize_t assign_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
+ /* If the guest is running, disallow assignment of domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -432,6 +451,10 @@ static ssize_t unassign_domain_store(struct device *dev,
struct mdev_device *mdev = mdev_from_dev(dev);
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ /* If the guest is running, disallow un-assignment of domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &apqi);
if (ret)
return ret;
@@ -470,6 +493,10 @@ static ssize_t assign_control_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long maxid = matrix_mdev->matrix.adm_max;
+ /* If the guest is running, disallow assignment of control domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &id);
if (ret)
return ret;
@@ -514,6 +541,10 @@ static ssize_t unassign_control_domain_store(struct device *dev,
struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
unsigned long max_domid = matrix_mdev->matrix.adm_max;
+ /* If the guest is running, disallow un-assignment of control domain */
+ if (matrix_mdev->kvm)
+ return -EBUSY;
+
ret = kstrtoul(buf, 0, &domid);
if (ret)
return ret;
@@ -602,7 +633,6 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
}
DEVICE_ATTR_RO(matrix);
-
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
@@ -624,12 +654,155 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
NULL
};
+/**
+ * Verify that the AP instructions are available on the guest and are to be
+ * interpreted by the firmware. The former is indicated via the
+ * KVM_S390_VM_CPU_FEAT_AP CPU model feature and the latter by apie crypto
+ * flag.
+ */
+static int kvm_ap_validate_crypto_setup(struct kvm *kvm)
+{
+ if (test_bit_inv(KVM_S390_VM_CPU_FEAT_AP, kvm->arch.cpu_feat) &&
+ kvm->arch.crypto.apie)
+ return 0;
+
+ return -EOPNOTSUPP;
+}
+
+static void kvm_ap_merge_bitmasks(unsigned long *dst, unsigned long *mask1,
+ unsigned long *mask2, unsigned long nbits)
+{
+ int i;
+
+ for (i = 0; i < BITS_TO_LONGS(nbits); i++)
+ dst[i] = mask1[i] | mask2[i];
+}
+
+static void vfio_ap_mdev_copy_masks(struct ap_matrix_mdev *matrix_mdev)
+{
+ int nbytes;
+ unsigned long *apm, *aqm, *adm;
+ struct kvm_s390_crypto_cb *crycb = matrix_mdev->kvm->arch.crypto.crycb;
+
+ switch (matrix_mdev->kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK) {
+ case CRYCB_FORMAT2:
+ apm = (unsigned long *)crycb->apcb1.apm;
+ aqm = (unsigned long *)crycb->apcb1.aqm;
+ adm = (unsigned long *)crycb->apcb1.adm;
+ break;
+ case CRYCB_FORMAT1:
+ case CRYCB_FORMAT0:
+ default:
+ apm = (unsigned long *)crycb->apcb0.apm;
+ aqm = (unsigned long *)crycb->apcb0.aqm;
+ adm = (unsigned long *)crycb->apcb0.adm;
+ break;
+ }
+
+ nbytes = DIV_ROUND_UP(matrix_mdev->matrix.apm_max + 1, BITS_PER_BYTE);
+ memcpy(apm, matrix_mdev->matrix.apm, nbytes);
+ nbytes = DIV_ROUND_UP(matrix_mdev->matrix.aqm_max + 1, BITS_PER_BYTE);
+ memcpy(aqm, matrix_mdev->matrix.aqm, nbytes);
+ kvm_ap_merge_bitmasks(adm, aqm, adm, matrix_mdev->matrix.adm_max + 1);
+}
+
+/**
+ * vfio_ap_mdev_open_once
+ *
+ * @matrix_mdev: a mediated matrix device
+ *
+ * Return 0 if no other mediated matrix device has been opened for the
+ * KVM guest assigned to @matrix_mdev; otherwise, returns an error.
+ */
+static int vfio_ap_mdev_open_once(struct ap_matrix_mdev *matrix_mdev,
+ struct kvm *kvm)
+{
+ struct ap_matrix_mdev *m;
+
+ mutex_lock(&matrix_dev.lock);
+
+ list_for_each_entry(m, &matrix_dev.mdev_list, list) {
+ if ((m != matrix_mdev) && (m->kvm == matrix_mdev->kvm)) {
+ mutex_unlock(&matrix_dev.lock);
+ return -EPERM;
+ }
+ }
+
+ mutex_unlock(&matrix_dev.lock);
+
+ return 0;
+}
+
+static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ int ret;
+ struct ap_matrix_mdev *matrix_mdev;
+
+ if (action != VFIO_GROUP_NOTIFY_SET_KVM)
+ return NOTIFY_OK;
+
+ matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
+
+ matrix_mdev->kvm = data;
+ if (data == NULL)
+ return NOTIFY_OK;
+
+ ret = vfio_ap_mdev_open_once(matrix_mdev, data);
+ if (ret)
+ return ret;
+
+ ret = kvm_ap_validate_crypto_setup(matrix_mdev->kvm);
+ if (ret)
+ return ret;
+
+ vfio_ap_mdev_copy_masks(matrix_mdev);
+
+ return NOTIFY_OK;
+}
+
+static int vfio_ap_mdev_open(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long events;
+ int ret;
+
+
+ if (!try_module_get(THIS_MODULE))
+ return -ENODEV;
+
+ matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
+ events = VFIO_GROUP_NOTIFY_SET_KVM;
+
+ ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+ &events, &matrix_mdev->group_notifier);
+ if (ret) {
+ module_put(THIS_MODULE);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void vfio_ap_mdev_release(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+ vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
+ &matrix_mdev->group_notifier);
+ matrix_mdev->kvm = NULL;
+ module_put(THIS_MODULE);
+}
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
.mdev_attr_groups = vfio_ap_mdev_attr_groups,
.create = vfio_ap_mdev_create,
.remove = vfio_ap_mdev_remove,
+ .open = vfio_ap_mdev_open,
+ .release = vfio_ap_mdev_release,
};
int vfio_ap_mdev_register(void)
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 1724258..6fa77cf 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -66,6 +66,8 @@ struct ap_matrix_mdev {
const char *name;
struct list_head list;
struct ap_matrix matrix;
+ struct notifier_block group_notifier;
+ struct kvm *kvm;
};
static inline struct device *to_device(struct ap_matrix_dev *matrix_dev)
--
1.7.1
From: Tony Krowiak <[email protected]>
Introduces two new KVM interface to clear the APM, AQM and ADM masks in
the guest's CRYCB. The VCPUs are taken out of SIE to ensure the VCPUs do
not get out of sync.
Signed-off-by: Tony Krowiak <[email protected]>
Acked-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
arch/s390/include/asm/kvm_host.h | 3 +++
arch/s390/kvm/kvm-s390.c | 15 +++++++++++++++
2 files changed, 18 insertions(+), 0 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 0c13f61..d7e19e1 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -258,6 +258,7 @@ struct kvm_s390_sie_block {
__u64 tecmc; /* 0x00e8 */
__u8 reservedf0[12]; /* 0x00f0 */
#define CRYCB_FORMAT_MASK 0x00000003
+#define CRYCB_FORMAT0 0x00000000
#define CRYCB_FORMAT1 0x00000001
#define CRYCB_FORMAT2 0x00000003
__u32 crycbd; /* 0x00fc */
@@ -862,6 +863,8 @@ void kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
struct kvm_async_pf *work);
+void kvm_arch_crypto_clear_masks(struct kvm *kvm);
+
extern int sie64a(struct kvm_s390_sie_block *, u64 *);
extern char sie_exit;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index f96a443..0d03249 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -1919,6 +1919,21 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
}
+void kvm_arch_crypto_clear_masks(struct kvm *kvm)
+{
+ mutex_lock(&kvm->lock);
+ kvm_s390_vcpu_block_all(kvm);
+
+ memset(&kvm->arch.crypto.crycb->apcb0, 0,
+ sizeof(kvm->arch.crypto.crycb->apcb0));
+ memset(&kvm->arch.crypto.crycb->apcb1, 0,
+ sizeof(kvm->arch.crypto.crycb->apcb1));
+
+ kvm_s390_vcpu_unblock_all(kvm);
+ mutex_unlock(&kvm->lock);
+}
+EXPORT_SYMBOL(kvm_arch_crypto_clear_masks);
+
static u64 kvm_s390_get_initial_cpuid(void)
{
struct cpuid cpuid;
--
1.7.1
From: Tony Krowiak <[email protected]>
Registers the matrix device created by the VFIO AP device
driver with the VFIO mediated device framework.
Registering the matrix device will create the sysfs
structures needed to create mediated matrix devices
each of which will be used to configure the AP matrix
for a guest and connect it to the VFIO AP device driver.
Registering the matrix device with the VFIO mediated device
framework will create the following sysfs structures:
/sys/devices/virtual/misc/vfio_ap/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ create
To create a mediated device for the AP matrix device, write a UUID
to the create file:
uuidgen > create
A symbolic link to the mediated device's directory will be created in the
devices subdirectory named after the generated $uuid:
/sys/devices/virtual/misc/vfio_ap/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
............... [$uuid]
A symbolic link to the mediated device will also be created
in the vfio_ap matrix's directory:
/sys/devices/virtual/misc/vfio_ap/[$uuid]
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
MAINTAINERS | 1 +
drivers/s390/crypto/Makefile | 2 +-
drivers/s390/crypto/vfio_ap_drv.c | 22 ++++++
drivers/s390/crypto/vfio_ap_ops.c | 125 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 49 +++++++++++++
include/uapi/linux/vfio.h | 1 +
6 files changed, 199 insertions(+), 1 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_ops.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 0e0792a..3597910 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12427,6 +12427,7 @@ W: http://www.ibm.com/developerworks/linux/linux390/
S: Supported
F: drivers/s390/crypto/vfio_ap_drv.c
F: drivers/s390/crypto/vfio_ap_private.h
+F: drivers/s390/crypto/vfio_ap_ops.c
S390 ZFCP DRIVER
M: Steffen Maier <[email protected]>
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index 48e466e..8d36b05 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -17,5 +17,5 @@ pkey-objs := pkey_api.o
obj-$(CONFIG_PKEY) += pkey.o
# adjunct processor matrix
-vfio_ap-objs := vfio_ap_drv.o
+vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o
obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
index d7e39ad..6a827f3 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -12,6 +12,7 @@
#include <linux/mod_devicetable.h>
#include <linux/slab.h>
#include <linux/string.h>
+#include <asm/zcrypt.h>
#include "vfio_ap_private.h"
#define VFIO_AP_ROOT_NAME "vfio_ap"
@@ -68,6 +69,18 @@ static int vfio_ap_matrix_dev_create(void)
{
int ret;
+ mutex_init(&matrix_dev.lock);
+ INIT_LIST_HEAD(&matrix_dev.mdev_list);
+
+ /* Test if PQAP(QCI) instruction is available */
+ if (test_facility(12)) {
+ ret = ap_qci(&matrix_dev.info);
+ if (ret)
+ return ret;
+ }
+
+ atomic_set(&matrix_dev.available_instances, MAX_ZDEV_ENTRIES_EXT);
+
ret = misc_register(&matrix_dev.misc_dev);
if (ret)
return ret;
@@ -102,11 +115,20 @@ int __init vfio_ap_init(void)
return ret;
}
+ ret = vfio_ap_mdev_register();
+ if (ret) {
+ ap_driver_unregister(&vfio_ap_drv);
+ vfio_ap_matrix_dev_destroy();
+
+ return ret;
+ }
+
return 0;
}
void __exit vfio_ap_exit(void)
{
+ vfio_ap_mdev_unregister();
ap_driver_unregister(&vfio_ap_drv);
vfio_ap_matrix_dev_destroy();
}
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
new file mode 100644
index 0000000..2d2b6fe
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -0,0 +1,125 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Adjunct processor matrix VFIO device driver callbacks.
+ *
+ * Copyright IBM Corp. 2018
+ *
+ * Author(s): Tony Krowiak <[email protected]>
+ * Halil Pasic <[email protected]>
+ * Pierre Morel <[email protected]>
+ */
+#include <linux/string.h>
+#include <linux/vfio.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/ctype.h>
+
+#include "vfio_ap_private.h"
+
+#define VFIO_AP_MDEV_TYPE_HWVIRT "passthrough"
+#define VFIO_AP_MDEV_NAME_HWVIRT "VFIO AP Passthrough Device"
+
+static void vfio_ap_matrix_init(struct ap_config_info *info,
+ struct ap_matrix *matrix)
+{
+ matrix->apm_max = info->apxa ? info->Na : 63;
+ matrix->aqm_max = info->apxa ? info->Nd : 15;
+ matrix->adm_max = info->apxa ? info->Nd : 15;
+}
+
+static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev;
+
+ matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
+ if (!matrix_mdev)
+ return -ENOMEM;
+
+ matrix_mdev->name = dev_name(mdev_dev(mdev));
+ vfio_ap_matrix_init(&matrix_dev.info, &matrix_mdev->matrix);
+ mdev_set_drvdata(mdev, matrix_mdev);
+
+ if (atomic_dec_if_positive(&matrix_dev.available_instances) < 0) {
+ kfree(matrix_mdev);
+ return -EPERM;
+ }
+
+ mutex_lock(&matrix_dev.lock);
+ list_add(&matrix_mdev->list, &matrix_dev.mdev_list);
+ mutex_unlock(&matrix_dev.lock);
+
+ return 0;
+}
+
+static int vfio_ap_mdev_remove(struct mdev_device *mdev)
+{
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ mutex_lock(&matrix_dev.lock);
+ list_del(&matrix_mdev->list);
+ mutex_unlock(&matrix_dev.lock);
+ kfree(matrix_mdev);
+ mdev_set_drvdata(mdev, NULL);
+ atomic_inc(&matrix_dev.available_instances);
+
+ return 0;
+}
+
+static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT);
+}
+
+MDEV_TYPE_ATTR_RO(name);
+
+static ssize_t available_instances_show(struct kobject *kobj,
+ struct device *dev, char *buf)
+{
+ return sprintf(buf, "%d\n",
+ atomic_read(&matrix_dev.available_instances));
+}
+
+MDEV_TYPE_ATTR_RO(available_instances);
+
+static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
+ char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_DEVICE_API_AP_STRING);
+}
+
+MDEV_TYPE_ATTR_RO(device_api);
+
+static struct attribute *vfio_ap_mdev_type_attrs[] = {
+ &mdev_type_attr_name.attr,
+ &mdev_type_attr_device_api.attr,
+ &mdev_type_attr_available_instances.attr,
+ NULL,
+};
+
+static struct attribute_group vfio_ap_mdev_hwvirt_type_group = {
+ .name = VFIO_AP_MDEV_TYPE_HWVIRT,
+ .attrs = vfio_ap_mdev_type_attrs,
+};
+
+static struct attribute_group *vfio_ap_mdev_type_groups[] = {
+ &vfio_ap_mdev_hwvirt_type_group,
+ NULL,
+};
+
+static const struct mdev_parent_ops vfio_ap_matrix_ops = {
+ .owner = THIS_MODULE,
+ .supported_type_groups = vfio_ap_mdev_type_groups,
+ .create = vfio_ap_mdev_create,
+ .remove = vfio_ap_mdev_remove,
+};
+
+int vfio_ap_mdev_register(void)
+{
+ return mdev_register_device(to_device(&matrix_dev),
+ &vfio_ap_matrix_ops);
+}
+
+void vfio_ap_mdev_unregister(void)
+{
+ mdev_unregister_device(to_device(&matrix_dev));
+}
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
index 02c878c..1724258 100644
--- a/drivers/s390/crypto/vfio_ap_private.h
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -3,6 +3,9 @@
* Private data and functions for adjunct processor VFIO matrix driver.
*
* Copyright IBM Corp. 2018
+ *
+ * Author(s): Tony Krowiak <[email protected]>
+ * Halil Pasic <[email protected]>
*/
#ifndef _VFIO_AP_PRIVATE_H_
@@ -10,6 +13,10 @@
#include <linux/types.h>
#include <linux/miscdevice.h>
+#include <linux/device.h>
+#include <linux/mdev.h>
+#include <linux/delay.h>
+#include <linux/mutex.h>
#include "ap_bus.h"
@@ -18,13 +25,55 @@
struct ap_matrix_dev {
struct miscdevice misc_dev;
+ atomic_t available_instances;
+ struct ap_config_info info;
+ struct list_head mdev_list;
+ struct mutex lock;
};
+/**
+ * Locking strategy: take the matrix_dev.lock mutex each time we fiddle
+ * with state managed by the vfio_ap driver (be it using the
+ * mdev_list or be it reading or writing the state of a single
+ * ap_matrix_mdev device). It's quite coarse but we don't expect
+ * much contention.
+ */
extern struct ap_matrix_dev matrix_dev;
+/**
+ * The AP matrix is comprised of three bit masks identifying the adapters,
+ * queues (domains) and control domains that belong to an AP matrix. The bits in
+ * each mask, from least significant to most significant bit, correspond to IDs
+ * 0 to 255. When a bit is set, the corresponding ID belongs to the matrix.
+ *
+ * @apm_max: max adapter number in @apm
+ * @apm identifies the AP adapters in the matrix
+ * @aqm_max: max domain number in @aqm
+ * @aqm identifies the AP queues (domains) in the matrix
+ * @adm_max: max domain number in @adm
+ * @adm identifies the AP control domains in the matrix
+ */
+struct ap_matrix {
+ unsigned long apm_max;
+ DECLARE_BITMAP(apm, 256);
+ unsigned long aqm_max;
+ DECLARE_BITMAP(aqm, 256);
+ unsigned long adm_max;
+ DECLARE_BITMAP(adm, 256);
+};
+
+struct ap_matrix_mdev {
+ const char *name;
+ struct list_head list;
+ struct ap_matrix matrix;
+};
+
static inline struct device *to_device(struct ap_matrix_dev *matrix_dev)
{
return matrix_dev->misc_dev.this_device;
}
+extern int vfio_ap_mdev_register(void);
+extern void vfio_ap_mdev_unregister(void);
+
#endif /* _VFIO_AP_PRIVATE_H_ */
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 1aa7b82..bfbe2be 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -215,6 +215,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform"
#define VFIO_DEVICE_API_AMBA_STRING "vfio-amba"
#define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"
+#define VFIO_DEVICE_API_AP_STRING "vfio-ap"
/**
* VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
--
1.7.1
From: Tony Krowiak <[email protected]>
Provides a sysfs interface to view the AP matrix configured for the
mediated matrix device.
The relevant sysfs structures are:
/sys/devices/virtual/misc/vfio_ap/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. matrix
To view the matrix configured for the mediated matrix device,
print the matrix file:
cat matrix
Below are examples of the output from the above command:
Example 1: Adapters and domains assigned
Assignments:
Adapters 5 and 6
Domains 4 and 71 (0x47)
Output
05.0004
05.0047
06.0004
06.0047
Examples 2: Only adapters assigned
Assignments:
Adapters 5 and 6
Output:
05.
06.
Examples 3: Only domains assigned
Assignments:
Domains 4 and 71 (0x47)
Output:
.0004
.0047
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 52 +++++++++++++++++++++++++++++++++++++
1 files changed, 52 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index a48676a..61a25d4 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -552,6 +552,57 @@ static ssize_t control_domains_show(struct device *dev,
}
DEVICE_ATTR_RO(control_domains);
+static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ char *bufpos = buf;
+ unsigned long apid;
+ unsigned long apqi;
+ unsigned long apid1;
+ unsigned long apqi1;
+ unsigned long napm_bits = matrix_mdev->matrix.apm_max + 1;
+ unsigned long naqm_bits = matrix_mdev->matrix.aqm_max + 1;
+ int nchars = 0;
+ int n;
+
+ apid1 = find_first_bit_inv(matrix_mdev->matrix.apm, napm_bits);
+ apqi1 = find_first_bit_inv(matrix_mdev->matrix.aqm, naqm_bits);
+
+ mutex_lock(&matrix_dev.lock);
+
+ if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) {
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+ naqm_bits) {
+ n = sprintf(bufpos, "%02lx.%04lx\n", apid,
+ apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+ } else if (apid1 < napm_bits) {
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
+ n = sprintf(bufpos, "%02lx.\n", apid);
+ bufpos += n;
+ nchars += n;
+ }
+ } else if (apqi1 < naqm_bits) {
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, naqm_bits) {
+ n = sprintf(bufpos, ".%04lx\n", apqi);
+ bufpos += n;
+ nchars += n;
+ }
+ }
+
+ mutex_unlock(&matrix_dev.lock);
+
+ return nchars;
+}
+DEVICE_ATTR_RO(matrix);
+
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
@@ -560,6 +611,7 @@ static ssize_t control_domains_show(struct device *dev,
&dev_attr_assign_control_domain.attr,
&dev_attr_unassign_control_domain.attr,
&dev_attr_control_domains.attr,
+ &dev_attr_matrix.attr,
NULL,
};
--
1.7.1
From: Tony Krowiak <[email protected]>
Introduces two new sysfs attributes for the VFIO mediated
matrix device for assigning AP domains to and unassigning
AP domains from a mediated matrix device. The IDs of the
AP domains assigned to the mediated matrix device will be
stored in an AP queue mask (AQM).
The bits in the AQM, from most significant to least
significant bit, correspond to AP queue index (APQI) 0 to
255 (note that an APQI is synonymous with with a domain ID).
On some systems, the maximum allowable domain number may be
less than 255 - depending upon the host's AP configuration -
and assignment may be rejected if the input domain ID exceeds
the limit.
When a domain is assigned, the bit corresponding to the APQI
will be set in the AQM. Likewise, when a domain is unassigned,
the bit corresponding to the APQI will be cleared from the AQM.
In order to successfully assign a domain, the APQNs derived from
the domain ID being assigned and the adapter numbers of all
adapters previously assigned:
1. Must be bound to the vfio_ap device driver.
2. Must not be assigned to any other mediated matrix device.
If there are no adapters assigned to the mdev, then there must
be an AP queue bound to the vfio_ap device driver with an
APQN containing the domain ID (i.e., APQI), otherwise all
adapters subsequently assigned will fail because there will be no
AP queues bound with an APQN containing the APQI.
Assigning or un-assigning an AP domain will also be rejected if
a guest using the mediated matrix device is running.
The relevant sysfs structures are:
/sys/devices/virtual/misc/vfio_ap/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_domain
.................. unassign_domain
To assign a domain to the $uuid mediated matrix device,
write the domain's ID to the assign_domain file. To
unassign a domain, write the domain's ID to the
unassign_domain file. The ID is specified using
conventional semantics: If it begins with 0x, the number
will be parsed as a hexadecimal (case insensitive) number;
if it begins with 0, it will be parsed as an octal number;
otherwise, it will be parsed as a decimal number.
For example, to assign domain 173 (0xad) to the mediated matrix
device $uuid:
echo 173 > assign_domain
or
echo 0255 > assign_domain
or
echo 0xad > assign_domain
To unassign domain 173 (0xad):
echo 173 > unassign_domain
or
echo 0255 > unassign_domain
or
echo 0xad > unassign_domain
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 88 ++++++++++++++++++++++++++++++++++++-
1 files changed, 87 insertions(+), 1 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 0a9f61f..36b8f47 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -363,10 +363,96 @@ static ssize_t unassign_adapter_store(struct device *dev,
}
DEVICE_ATTR_WO(unassign_adapter);
+static int
+vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apqi)
+{
+ int ret;
+ unsigned long apid;
+ unsigned long nbits = matrix_mdev->matrix.apm_max + 1;
+
+ if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits)
+ return vfio_ap_verify_queue_reserved(NULL, &apqi);
+
+ for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) {
+ ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static ssize_t assign_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret)
+ return ret;
+ if (apqi > max_apqi)
+ return -EINVAL;
+
+ mutex_lock(&matrix_dev.lock);
+
+ ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi);
+ if (ret)
+ goto done;
+
+ set_bit_inv(apqi, matrix_mdev->matrix.aqm);
+
+ ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
+ if (ret)
+ goto share_err;
+
+ ret = count;
+ goto done;
+
+share_err:
+ clear_bit_inv(apqi, matrix_mdev->matrix.aqm);
+done:
+ mutex_unlock(&matrix_dev.lock);
+
+ return ret;
+}
+DEVICE_ATTR_WO(assign_domain);
+
+static ssize_t unassign_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apqi;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ ret = kstrtoul(buf, 0, &apqi);
+ if (ret)
+ return ret;
+
+ if (apqi > matrix_mdev->matrix.aqm_max)
+ return -EINVAL;
+
+ mutex_lock(&matrix_dev.lock);
+ clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
+ mutex_unlock(&matrix_dev.lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_domain);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
- NULL
+ &dev_attr_assign_domain.attr,
+ &dev_attr_unassign_domain.attr,
+ NULL,
};
static struct attribute_group vfio_ap_mdev_attr_group = {
--
1.7.1
From: Tony Krowiak <[email protected]>
Provides the sysfs interfaces for:
1. Assigning AP control domains to the mediated matrix device
2. Unassigning AP control domains from a mediated matrix device
3. Displaying the control domains assigned to a mediated matrix
device
The IDs of the AP control domains assigned to the mediated matrix
device are stored in an AP domain mask (ADM). The bits in the ADM,
from most significant to least significant bit, correspond to
AP domain numbers 0 to 255. On some systems, the maximum allowable
domain number may be less than 255 - depending upon the host's
AP configuration - and assignment may be rejected if the input
domain ID exceeds the limit.
When a control domain is assigned, the bit corresponding its domain
ID will be set in the ADM. Likewise, when a domain is unassigned,
the bit corresponding to its domain ID will be cleared in the ADM.
The relevant sysfs structures are:
/sys/devices/virtual/misc/vfio_ap/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_control_domain
.................. unassign_control_domain
To assign a control domain to the $uuid mediated matrix device's
ADM, write its domain number to the assign_control_domain file.
To unassign a domain, write its domain number to the
unassign_control_domain file. The domain number is specified
using conventional semantics: If it begins with 0x the number
will be parsed as a hexadecimal (case insensitive) number;
if it begins with 0, it is parsed as an octal number;
otherwise, it will be parsed as a decimal number.
For example, to assign control domain 173 (0xad) to the mediated
matrix device $uuid:
echo 173 > assign_control_domain
or
echo 0255 > assign_control_domain
or
echo 0xad > assign_control_domain
To unassign control domain 173 (0xad):
echo 173 > unassign_control_domain
or
echo 0255 > unassign_control_domain
or
echo 0xad > unassign_control_domain
The assignment will be rejected if the APQI exceeds the maximum
value for an AP domain:
* If the AP Extended Addressing (APXA) facility is installed,
the max value is 255
* Else the max value is 15
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 108 +++++++++++++++++++++++++++++++++++++
1 files changed, 108 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 36b8f47..a48676a 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -447,11 +447,119 @@ static ssize_t unassign_domain_store(struct device *dev,
}
DEVICE_ATTR_WO(unassign_domain);
+/**
+ * assign_control_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's assign_control_domain attribute
+ * @buf: a buffer containing the domain ID to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the domain ID from @buf and assigns it to the mediated matrix device.
+ *
+ * Returns the number of bytes processed if the domain ID is valid; otherwise
+ * returns an error.
+ */
+static ssize_t assign_control_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long id;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long maxid = matrix_mdev->matrix.adm_max;
+
+ ret = kstrtoul(buf, 0, &id);
+ if (ret)
+ return ret;
+
+ if (id > maxid)
+ ret = -EINVAL;
+
+ /* Set the bit in the ADM (bitmask) corresponding to the AP control
+ * domain number (id). The bits in the mask, from most significant to
+ * least significant, correspond to IDs 0 up to the one less than the
+ * number of control domains that can be assigned.
+ */
+ mutex_lock(&matrix_dev.lock);
+ set_bit_inv(id, matrix_mdev->matrix.adm);
+ mutex_unlock(&matrix_dev.lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(assign_control_domain);
+
+/**
+ * unassign_control_domain_store
+ *
+ * @dev: the matrix device
+ * @attr: the mediated matrix device's unassign_control_domain attribute
+ * @buf: a buffer containing the domain ID to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the domain ID from @buf and unassigns it from the mediated matrix
+ * device.
+ *
+ * Returns the number of bytes processed if the domain ID is valid; otherwise
+ * returns an error.
+ */
+static ssize_t unassign_control_domain_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long domid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_domid = matrix_mdev->matrix.adm_max;
+
+ ret = kstrtoul(buf, 0, &domid);
+ if (ret)
+ return ret;
+ if (domid > max_domid)
+ return -EINVAL;
+
+ mutex_lock(&matrix_dev.lock);
+ clear_bit_inv(domid, matrix_mdev->matrix.adm);
+ mutex_unlock(&matrix_dev.lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_control_domain);
+
+static ssize_t control_domains_show(struct device *dev,
+ struct device_attribute *dev_attr,
+ char *buf)
+{
+ unsigned long id;
+ int nchars = 0;
+ int n;
+ char *bufpos = buf;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_domid = matrix_mdev->matrix.adm_max;
+
+ mutex_lock(&matrix_dev.lock);
+ for_each_set_bit_inv(id, matrix_mdev->matrix.adm, max_domid + 1) {
+ n = sprintf(bufpos, "%04lx\n", id);
+ bufpos += n;
+ nchars += n;
+ }
+ mutex_unlock(&matrix_dev.lock);
+
+ return nchars;
+}
+DEVICE_ATTR_RO(control_domains);
+
static struct attribute *vfio_ap_mdev_attrs[] = {
&dev_attr_assign_adapter.attr,
&dev_attr_unassign_adapter.attr,
&dev_attr_assign_domain.attr,
&dev_attr_unassign_domain.attr,
+ &dev_attr_assign_control_domain.attr,
+ &dev_attr_unassign_control_domain.attr,
+ &dev_attr_control_domains.attr,
NULL,
};
--
1.7.1
From: David Hildenbrand <[email protected]>
When we change the crycb (or execution controls), we also have to make sure
that the vSIE shadow datastructures properly consider the changed
values before rerunning the vSIE. We can achieve that by simply using a
VCPU request now.
This has to be a synchronous request (== handled before entering the
(v)SIE again).
The request will make sure that the vSIE handler is left, and that the
request will be processed (NOP), therefore forcing a reload of all
vSIE data (including rebuilding the crycb) when re-entering the vSIE
interception handler the next time.
Signed-off-by: David Hildenbrand <[email protected]>
Reviewed-by: Pierre Morel<[email protected]>
---
arch/s390/include/asm/kvm_host.h | 1 +
arch/s390/kvm/kvm-s390.c | 7 ++++++-
2 files changed, 7 insertions(+), 1 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index a2188e3..af39561 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -44,6 +44,7 @@
#define KVM_REQ_ICPT_OPEREXC KVM_ARCH_REQ(2)
#define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3)
#define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4)
+#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5)
#define SIGP_CTRL_C 0x80
#define SIGP_CTRL_SCN_MASK 0x3f
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 6df2d12..19f4f44 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -798,8 +798,11 @@ void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm)
kvm_s390_vcpu_block_all(kvm);
- kvm_for_each_vcpu(i, vcpu, kvm)
+ kvm_for_each_vcpu(i, vcpu, kvm) {
kvm_s390_vcpu_crypto_setup(vcpu);
+ /* recreate the shadow crycb by leaving the VSIE handler */
+ kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
+ }
kvm_s390_vcpu_unblock_all(kvm);
}
@@ -3090,6 +3093,8 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
/* nothing to do, just clear the request */
kvm_clear_request(KVM_REQ_UNHALT, vcpu);
+ /* we left the vsie handler, nothing to do, just clear the request */
+ kvm_clear_request(KVM_REQ_VSIE_RESTART, vcpu);
return 0;
}
--
1.7.1
From: Harald Freudenberger <[email protected]>
Show the current load value of cards and queues in sysfs.
The load value for each card and queue is maintained by
the zcrypt device driver for dispatching and load
balancing requests over the available devices.
This patch provides the load value to userspace via a
new read only sysfs attribute 'load' per card and queue.
Signed-off-by: Harald Freudenberger <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/zcrypt_card.c | 12 ++++++++++++
drivers/s390/crypto/zcrypt_queue.c | 12 ++++++++++++
2 files changed, 24 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/zcrypt_card.c b/drivers/s390/crypto/zcrypt_card.c
index 233e1e6..da2c8df 100644
--- a/drivers/s390/crypto/zcrypt_card.c
+++ b/drivers/s390/crypto/zcrypt_card.c
@@ -83,9 +83,21 @@ static ssize_t zcrypt_card_online_store(struct device *dev,
static DEVICE_ATTR(online, 0644, zcrypt_card_online_show,
zcrypt_card_online_store);
+static ssize_t zcrypt_card_load_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct zcrypt_card *zc = to_ap_card(dev)->private;
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", atomic_read(&zc->load));
+}
+
+static DEVICE_ATTR(load, 0444, zcrypt_card_load_show, NULL);
+
static struct attribute *zcrypt_card_attrs[] = {
&dev_attr_type.attr,
&dev_attr_online.attr,
+ &dev_attr_load.attr,
NULL,
};
diff --git a/drivers/s390/crypto/zcrypt_queue.c b/drivers/s390/crypto/zcrypt_queue.c
index 720434e..91a52f2 100644
--- a/drivers/s390/crypto/zcrypt_queue.c
+++ b/drivers/s390/crypto/zcrypt_queue.c
@@ -75,8 +75,20 @@ static ssize_t zcrypt_queue_online_store(struct device *dev,
static DEVICE_ATTR(online, 0644, zcrypt_queue_online_show,
zcrypt_queue_online_store);
+static ssize_t zcrypt_queue_load_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct zcrypt_queue *zq = to_ap_queue(dev)->private;
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", atomic_read(&zq->load));
+}
+
+static DEVICE_ATTR(load, 0444, zcrypt_queue_load_show, NULL);
+
static struct attribute *zcrypt_queue_attrs[] = {
&dev_attr_online.attr,
+ &dev_attr_load.attr,
NULL,
};
--
1.7.1
From: Tony Krowiak <[email protected]>
Introduces two new sysfs attributes for the VFIO mediated
matrix device for assigning AP adapters to and unassigning
AP adapters from a mediated matrix device. The IDs of the
AP adapters assigned to the mediated matrix device will be
stored in an AP mask (APM).
The bits in the APM, from most significant to least significant
bit, correspond to AP adapter IDs (APID) 0 to 255. On
some systems, the maximum allowable adapter number may be less
than 255 - depending upon the host's AP configuration - and
assignment may be rejected if the input adapter ID exceeds the
limit.
When an adapter is assigned, the bit corresponding to the APID
will be set in the APM. Likewise, when an adapter is
unassigned, the bit corresponding to the APID will be cleared
from the APM.
In order to successfully assign an adapter, the APQNs derived from
the adapter ID being assigned and the queue indexes of all domains
previously assigned:
1. Must be bound to the vfio_ap device driver.
2. Must not be assigned to any other mediated matrix device
If there are no domains assigned to the mdev, then there must
be an AP queue bound to the vfio_ap device driver with an
APQN containing the APID, otherwise all domains
subsequently assigned will fail because there will be no
AP queues bound with an APQN containing the adapter ID.
Assigning or un-assigning an AP adapter will be rejected if
a guest using the mediated matrix device is running.
The relevant sysfs structures are:
/sys/devices/virtual/misc/vfio_ap/
...... [mdev_supported_types]
......... [vfio_ap-passthrough]
............ [devices]
...............[$uuid]
.................. assign_adapter
.................. unassign_adapter
To assign an adapter to the $uuid mediated matrix device's APM,
write the APID to the assign_adapter file. To unassign an adapter,
write the APID to the unassign_adapter file. The APID is specified
using conventional semantics: If it begins with 0x the number will
be parsed as a hexadecimal number; if it begins with a 0 the number
will be parsed as an octal number; otherwise, it will be parsed as a
decimal number.
For example, to assign adapter 173 (0xad) to the mediated matrix
device $uuid:
echo 173 > assign_adapter
or
echo 0xad > assign_adapter
or
echo 0255 > assign_adapter
To unassign adapter 173 (0xad):
echo 173 > unassign_adapter
or
echo 0xad > unassign_adapter
or
echo 0255 > unassign_adapter
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Tested-by: Pierre Morel <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
drivers/s390/crypto/vfio_ap_ops.c | 273 +++++++++++++++++++++++++++++++++++++
1 files changed, 273 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 2d2b6fe..0a9f61f 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -106,9 +106,282 @@ static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
NULL,
};
+struct vfio_ap_queue_reserved {
+ unsigned long *apid;
+ unsigned long *apqi;
+ bool reserved;
+};
+
+/**
+ * vfio_ap_has_queue
+ *
+ * @dev: an AP queue device
+ * @data: a struct vfio_ap_queue_reserved reference
+ *
+ * Flags whether the AP queue device (@dev) has a queue ID containing the APQN,
+ * apid or apqi specified in @data:
+ *
+ * - If @data contains both an apid and apqi value, then @data will be flagged
+ * as reserved if the APID and APQI fields for the AP queue device matches
+ *
+ * - If @data contains only an apid value, @data will be flagged as
+ * reserved if the APID field in the AP queue device matches
+ *
+ * - If @data contains only an apqi value, @data will be flagged as
+ * reserved if the APQI field in the AP queue device matches
+ *
+ * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if
+ * @data does not contain either an apid or apqi.
+ */
+static int vfio_ap_has_queue(struct device *dev, void *data)
+{
+ struct vfio_ap_queue_reserved *qres = data;
+ struct ap_queue *ap_queue = to_ap_queue(dev);
+ ap_qid_t qid;
+ unsigned long id;
+
+ if (qres->apid && qres->apqi) {
+ qid = AP_MKQID(*qres->apid, *qres->apqi);
+ if (qid == ap_queue->qid)
+ qres->reserved = true;
+ } else if (qres->apid && !qres->apqi) {
+ id = AP_QID_CARD(ap_queue->qid);
+ if (id == *qres->apid)
+ qres->reserved = true;
+ } else if (!qres->apid && qres->apqi) {
+ id = AP_QID_QUEUE(ap_queue->qid);
+ if (id == *qres->apqi)
+ qres->reserved = true;
+ } else {
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+/**
+ * vfio_ap_verify_queue_reserved
+ *
+ * @matrix_dev: a mediated matrix device
+ * @apid: an AP adapter ID
+ * @apqi: an AP queue index
+ *
+ * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device
+ * driver according to the following rules:
+ *
+ * - If both @apid and @apqi are not NULL, then there must be an AP queue
+ * device bound to the vfio_ap driver with the APQN identified by @apid and
+ * @apqi
+ *
+ * - If only @apid is not NULL, then there must be an AP queue device bound
+ * to the vfio_ap driver with an APQN containing @apid
+ *
+ * - If only @apqi is not NULL, then there must be an AP queue device bound
+ * to the vfio_ap driver with an APQN containing @apqi
+ *
+ * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL.
+ */
+static int vfio_ap_verify_queue_reserved(unsigned long *apid,
+ unsigned long *apqi)
+{
+ int ret;
+ struct vfio_ap_queue_reserved qres;
+
+ qres.apid = apid;
+ qres.apqi = apqi;
+ qres.reserved = false;
+
+ ret = driver_for_each_device(matrix_dev.misc_dev.this_device->driver,
+ NULL, &qres, vfio_ap_has_queue);
+ if (ret)
+ return ret;
+
+ if (qres.reserved)
+ return 0;
+
+ return -EADDRNOTAVAIL;
+}
+
+static int
+vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev,
+ unsigned long apid)
+{
+ int ret;
+ unsigned long apqi;
+ unsigned long nbits = matrix_mdev->matrix.aqm_max + 1;
+
+ if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits)
+ return vfio_ap_verify_queue_reserved(&apid, NULL);
+
+ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) {
+ ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * vfio_ap_mdev_verify_no_sharing
+ *
+ * Verifies that the APQNs derived from the cross product of the AP adapter IDs
+ * and AP queue indexes comprising the AP matrix are not configured for another
+ * activated mediated device. AP queue sharing is not allowed.
+ *
+ * @kvm: the KVM guest
+ * @matrix: the AP matrix
+ *
+ * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
+ */
+static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+{
+ int nbits;
+ struct ap_matrix_mdev *lstdev;
+ unsigned long apm[BITS_TO_LONGS(matrix_mdev->matrix.apm_max + 1)];
+ unsigned long aqm[BITS_TO_LONGS(matrix_mdev->matrix.aqm_max + 1)];
+
+ list_for_each_entry(lstdev, &matrix_dev.mdev_list, list) {
+ if (matrix_mdev == lstdev)
+ continue;
+
+ memset(apm, 0, sizeof(apm));
+ memset(aqm, 0, sizeof(aqm));
+
+ /*
+ * We work on full longs, as we can only exclude the leftover
+ * bits in non-inverse order. The leftover is all zeros.
+ */
+ nbits = sizeof(apm) * BITS_PER_BYTE;
+ if (!bitmap_and(apm, matrix_mdev->matrix.apm,
+ lstdev->matrix.apm, nbits))
+ continue;
+
+ nbits = sizeof(aqm) * BITS_PER_BYTE;
+ if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
+ lstdev->matrix.aqm, nbits))
+ continue;
+
+ return -EADDRINUSE;
+ }
+
+ return 0;
+}
+
+/**
+ * assign_adapter_store
+ *
+ * @dev: the matrix device
+ * @attr: a mediated matrix device attribute
+ * @buf: a buffer containing the adapter ID (APID) to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APID from @buf and assigns it to the mediated matrix device.
+ *
+ * Returns the number of bytes processed if the APID is valid; otherwise returns
+ * an error.
+ */
+static ssize_t assign_adapter_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret = 0;
+ unsigned long apid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+ unsigned long max_apid = matrix_mdev->matrix.apm_max;
+
+ ret = kstrtoul(buf, 0, &apid);
+ if (ret)
+ return ret;
+ if (apid > max_apid)
+ return -EINVAL;
+
+ /* Set the bit in the AP mask (APM) corresponding to the AP adapter
+ * number (APID). The bits in the mask, from most significant to least
+ * significant bit, correspond to APIDs 0-255.
+ */
+ mutex_lock(&matrix_dev.lock);
+
+ ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid);
+ if (ret)
+ goto done;
+
+ set_bit_inv(apid, matrix_mdev->matrix.apm);
+
+ ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
+ if (ret)
+ goto share_err;
+
+ ret = count;
+ goto done;
+
+share_err:
+ clear_bit_inv(apid, matrix_mdev->matrix.apm);
+done:
+ mutex_unlock(&matrix_dev.lock);
+
+ return ret;
+}
+static DEVICE_ATTR_WO(assign_adapter);
+
+/**
+ * unassign_adapter_store
+ *
+ * @dev: the matrix device
+ * @attr: a mediated matrix device attribute
+ * @buf: a buffer containing the adapter ID (APID) to be assigned
+ * @count: the number of bytes in @buf
+ *
+ * Parses the APID from @buf and unassigns it from the mediated matrix device.
+ * The APID must be a valid value
+ *
+ * Returns the number of bytes processed if the APID is valid; otherwise returns
+ * an error.
+ */
+static ssize_t unassign_adapter_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret;
+ unsigned long apid;
+ struct mdev_device *mdev = mdev_from_dev(dev);
+ struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+ ret = kstrtoul(buf, 0, &apid);
+ if (ret)
+ return ret;
+
+ if (apid > matrix_mdev->matrix.apm_max)
+ return -EINVAL;
+
+ mutex_lock(&matrix_dev.lock);
+ clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
+ mutex_unlock(&matrix_dev.lock);
+
+ return count;
+}
+DEVICE_ATTR_WO(unassign_adapter);
+
+static struct attribute *vfio_ap_mdev_attrs[] = {
+ &dev_attr_assign_adapter.attr,
+ &dev_attr_unassign_adapter.attr,
+ NULL
+};
+
+static struct attribute_group vfio_ap_mdev_attr_group = {
+ .attrs = vfio_ap_mdev_attrs
+};
+
+static const struct attribute_group *vfio_ap_mdev_attr_groups[] = {
+ &vfio_ap_mdev_attr_group,
+ NULL
+};
+
static const struct mdev_parent_ops vfio_ap_matrix_ops = {
.owner = THIS_MODULE,
.supported_type_groups = vfio_ap_mdev_type_groups,
+ .mdev_attr_groups = vfio_ap_mdev_attr_groups,
.create = vfio_ap_mdev_create,
.remove = vfio_ap_mdev_remove,
};
--
1.7.1
From: David Hildenbrand <[email protected]>
VCPU requests and VCPU blocking right now don't take care of the vSIE
(as it was not necessary until now). But we want to have VCPU requests
that will also be handled before running the vSIE again.
So let's simulate a SIE entry when entering the vSIE loop and check
for PROG_ flags. The existing infrastructure (e.g. exit_sie()) will then
detect that the SIE (in form of the vSIE execution loop) is running and
properly kick the vSIE CPU, resulting in it leaving the vSIE loop and
therefore the vSIE interception handler, allowing it to handle VCPU
requests.
E.g. if we want to modify the crycb of the VCPU and make sure that any
masks also get applied to the VSIE crycb shadow (which uses masks from the
VCPU crycb), we will need a way to hinder the vSIE from running and make
sure to process the updated crycb before reentering the vSIE again.
Signed-off-by: David Hildenbrand <[email protected]>
Reviewed-by: Pierre Morel<[email protected]>
---
arch/s390/kvm/kvm-s390.c | 9 ++++++++-
arch/s390/kvm/kvm-s390.h | 1 +
arch/s390/kvm/vsie.c | 20 ++++++++++++++++++--
3 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 3b7a515..6df2d12 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2655,18 +2655,25 @@ static void kvm_s390_vcpu_request(struct kvm_vcpu *vcpu)
exit_sie(vcpu);
}
+bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu)
+{
+ return atomic_read(&vcpu->arch.sie_block->prog20) &
+ (PROG_BLOCK_SIE | PROG_REQUEST);
+}
+
static void kvm_s390_vcpu_request_handled(struct kvm_vcpu *vcpu)
{
atomic_andnot(PROG_REQUEST, &vcpu->arch.sie_block->prog20);
}
/*
- * Kick a guest cpu out of SIE and wait until SIE is not running.
+ * Kick a guest cpu out of (v)SIE and wait until (v)SIE is not running.
* If the CPU is not running (e.g. waiting as idle) the function will
* return immediately. */
void exit_sie(struct kvm_vcpu *vcpu)
{
kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOP_INT);
+ kvm_s390_vsie_kick(vcpu);
while (vcpu->arch.sie_block->prog0c & PROG_IN_SIE)
cpu_relax();
}
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index 981e3ba..1f6e36c 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -290,6 +290,7 @@ void kvm_s390_set_tod_clock(struct kvm *kvm,
void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu);
void kvm_s390_vcpu_block(struct kvm_vcpu *vcpu);
void kvm_s390_vcpu_unblock(struct kvm_vcpu *vcpu);
+bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu);
void exit_sie(struct kvm_vcpu *vcpu);
void kvm_s390_sync_request(int req, struct kvm_vcpu *vcpu);
int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 84c89cb..aa30b48 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -982,6 +982,17 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
int rc = 0;
+ /*
+ * Simulate a SIE entry of the VCPU (see sie64a), so VCPU blocking
+ * and VCPU requests can hinder the whole vSIE loop from running
+ * and lead to an immediate exit. We do it at this point (not
+ * earlier), so kvm_s390_vsie_kick() works correctly already.
+ */
+ vcpu->arch.sie_block->prog0c |= PROG_IN_SIE;
+ barrier();
+ if (kvm_s390_vcpu_sie_inhibited(vcpu))
+ return 0;
+
while (1) {
rc = acquire_gmap_shadow(vcpu, vsie_page);
if (!rc)
@@ -997,10 +1008,14 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
if (rc == -EAGAIN)
rc = 0;
if (rc || scb_s->icptcode || signal_pending(current) ||
- kvm_s390_vcpu_has_irq(vcpu, 0))
+ kvm_s390_vcpu_has_irq(vcpu, 0) ||
+ kvm_s390_vcpu_sie_inhibited(vcpu))
break;
}
+ barrier();
+ vcpu->arch.sie_block->prog0c &= ~PROG_IN_SIE;
+
if (rc == -EFAULT) {
/*
* Addressing exceptions are always presentes as intercepts.
@@ -1114,7 +1129,8 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu)
if (unlikely(scb_addr & 0x1ffUL))
return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
- if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0))
+ if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0) ||
+ kvm_s390_vcpu_sie_inhibited(vcpu))
return 0;
vsie_page = get_vsie_page(vcpu->kvm, scb_addr);
--
1.7.1
From: Tony Krowiak <[email protected]>
Introduces a new AP device driver. This device driver
is built on the VFIO mediated device framework. The framework
provides sysfs interfaces that facilitate passthrough
access by guests to devices installed on the linux host.
The VFIO AP device driver will serve two purposes:
1. Provide the interfaces to reserve AP devices for exclusive
use by KVM guests. This is accomplished by unbinding the
devices to be reserved for guest usage from the zcrypt
device driver and binding them to the VFIO AP device driver.
2. Implements the functions, callbacks and sysfs attribute
interfaces required to create one or more VFIO mediated
devices each of which will be used to configure the AP
matrix for a guest and serve as a file descriptor
for facilitating communication between QEMU and the
VFIO AP device driver.
When the VFIO AP device driver is initialized:
* It registers with the AP bus for control of type 10 (CEX4
and newer) AP queue devices. This limitation was imposed
due to:
1. A desire to keep the code as simple as possible;
2. Some older models are no longer supported by the kernel
and others are getting close to end of service.
The probe and remove callbacks will be provided to support
the binding/unbinding of AP queue devices to/from the VFIO
AP device driver.
* Creates a virtual misc device, /sys/devices/virtual/misc/vfio_ap,
device to serve as the parent of the mediated devices created, one
for each guest, and to hold the APQNs of the AP devices bound to
the VFIO AP device driver.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
MAINTAINERS | 10 +++
arch/s390/Kconfig | 11 +++
drivers/s390/crypto/Makefile | 4 +
drivers/s390/crypto/vfio_ap_drv.c | 115 +++++++++++++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_private.h | 30 +++++++++
5 files changed, 170 insertions(+), 0 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_drv.c
create mode 100644 drivers/s390/crypto/vfio_ap_private.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 7cebd5b..0e0792a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12418,6 +12418,16 @@ W: http://www.ibm.com/developerworks/linux/linux390/
S: Supported
F: drivers/s390/crypto/
+S390 VFIO AP DRIVER
+M: Tony Krowiak <[email protected]>
+M: Pierre Morel <[email protected]>
+M: Halil Pasic <[email protected]>
+L: [email protected]
+W: http://www.ibm.com/developerworks/linux/linux390/
+S: Supported
+F: drivers/s390/crypto/vfio_ap_drv.c
+F: drivers/s390/crypto/vfio_ap_private.h
+
S390 ZFCP DRIVER
M: Steffen Maier <[email protected]>
M: Benjamin Block <[email protected]>
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 8a1863d..35e9640 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -779,6 +779,17 @@ config VFIO_CCW
To compile this driver as a module, choose M here: the
module will be called vfio_ccw.
+config VFIO_AP
+ def_tristate n
+ prompt "VFIO support for AP devices"
+ depends on ZCRYPT && VFIO_MDEV_DEVICE && KVM
+ help
+ This driver grants access to Adjunct Processor (AP) devices
+ via the VFIO mediated device interface.
+
+ To compile this driver as a module, choose M here: the module
+ will be called vfio_ap.
+
endmenu
menu "Dump support"
diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile
index b59af54..48e466e 100644
--- a/drivers/s390/crypto/Makefile
+++ b/drivers/s390/crypto/Makefile
@@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o
# pkey kernel module
pkey-objs := pkey_api.o
obj-$(CONFIG_PKEY) += pkey.o
+
+# adjunct processor matrix
+vfio_ap-objs := vfio_ap_drv.o
+obj-$(CONFIG_VFIO_AP) += vfio_ap.o
diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
new file mode 100644
index 0000000..d7e39ad
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -0,0 +1,115 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * VFIO based AP device driver
+ *
+ * Copyright IBM Corp. 2018
+ *
+ * Author(s): Tony Krowiak <[email protected]>
+ * Halil Pasic <[email protected]>
+ */
+
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include "vfio_ap_private.h"
+
+#define VFIO_AP_ROOT_NAME "vfio_ap"
+#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
+#define VFIO_AP_DEV_NAME "vfio_ap"
+
+MODULE_AUTHOR("IBM Corporation");
+MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2018");
+MODULE_LICENSE("GPL v2");
+
+static struct ap_driver vfio_ap_drv;
+
+static struct device_type vfio_ap_dev_type = {
+ .name = VFIO_AP_DEV_TYPE_NAME,
+};
+
+struct ap_matrix_dev matrix_dev = {
+ /*
+ * We may add some fops later to implement a more program friendly
+ * management interface for vfio_ap_mdev devices.
+ */
+ .misc_dev = {
+ MISC_DYNAMIC_MINOR,
+ VFIO_AP_DEV_NAME,
+ }
+};
+
+/* Only type 10 adapters (CEX4 and later) are supported
+ * by the AP matrix device driver
+ */
+static struct ap_device_id ap_queue_ids[] = {
+ { .dev_type = AP_DEVICE_TYPE_CEX4,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { .dev_type = AP_DEVICE_TYPE_CEX5,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { .dev_type = AP_DEVICE_TYPE_CEX6,
+ .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
+ { /* end of sibling */ },
+};
+
+MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
+
+static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
+{
+ return 0;
+}
+
+static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
+{
+ /* Nothing to do yet */
+}
+
+static int vfio_ap_matrix_dev_create(void)
+{
+ int ret;
+
+ ret = misc_register(&matrix_dev.misc_dev);
+ if (ret)
+ return ret;
+
+ to_device(&matrix_dev)->type = &vfio_ap_dev_type;
+ to_device(&matrix_dev)->driver = &vfio_ap_drv.driver;
+
+ return ret;
+}
+
+static void vfio_ap_matrix_dev_destroy(void)
+{
+ misc_deregister(&matrix_dev.misc_dev);
+}
+
+int __init vfio_ap_init(void)
+{
+ int ret;
+
+ ret = vfio_ap_matrix_dev_create();
+ if (ret)
+ return ret;
+
+ memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
+ vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
+ vfio_ap_drv.remove = vfio_ap_queue_dev_remove;
+ vfio_ap_drv.ids = ap_queue_ids;
+
+ ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
+ if (ret) {
+ vfio_ap_matrix_dev_destroy();
+ return ret;
+ }
+
+ return 0;
+}
+
+void __exit vfio_ap_exit(void)
+{
+ ap_driver_unregister(&vfio_ap_drv);
+ vfio_ap_matrix_dev_destroy();
+}
+
+module_init(vfio_ap_init);
+module_exit(vfio_ap_exit);
diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
new file mode 100644
index 0000000..02c878c
--- /dev/null
+++ b/drivers/s390/crypto/vfio_ap_private.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Private data and functions for adjunct processor VFIO matrix driver.
+ *
+ * Copyright IBM Corp. 2018
+ */
+
+#ifndef _VFIO_AP_PRIVATE_H_
+#define _VFIO_AP_PRIVATE_H_
+
+#include <linux/types.h>
+#include <linux/miscdevice.h>
+
+#include "ap_bus.h"
+
+#define VFIO_AP_MODULE_NAME "vfio_ap"
+#define VFIO_AP_DRV_NAME "vfio_ap"
+
+struct ap_matrix_dev {
+ struct miscdevice misc_dev;
+};
+
+extern struct ap_matrix_dev matrix_dev;
+
+static inline struct device *to_device(struct ap_matrix_dev *matrix_dev)
+{
+ return matrix_dev->misc_dev.this_device;
+}
+
+#endif /* _VFIO_AP_PRIVATE_H_ */
--
1.7.1
From: Harald Freudenberger <[email protected]>
Reviewed and adapted the register use and asm constraints
of the C inline assembler functions in accordance to the
the AP instructions specifications.
Signed-off-by: Harald Freudenberger <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
Reviewed-by: Tony Krowiak <[email protected]>
---
drivers/s390/crypto/ap_asm.h | 40 +++++++++++++++++++++++-----------------
1 files changed, 23 insertions(+), 17 deletions(-)
diff --git a/drivers/s390/crypto/ap_asm.h b/drivers/s390/crypto/ap_asm.h
index b22d30a..e22ee12 100644
--- a/drivers/s390/crypto/ap_asm.h
+++ b/drivers/s390/crypto/ap_asm.h
@@ -20,14 +20,16 @@ static inline int ap_instructions_available(void)
{
register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
register unsigned long reg1 asm ("1") = -ENODEV;
- register unsigned long reg2 asm ("2") = 0UL;
+ register unsigned long reg2 asm ("2");
asm volatile(
" .long 0xb2af0000\n" /* PQAP(TAPQ) */
- "0: la %1,0\n"
+ "0: la %0,0\n"
"1:\n"
EX_TABLE(0b, 1b)
- : "+d" (reg0), "+d" (reg1), "+d" (reg2) : : "cc");
+ : "+d" (reg1), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
return reg1;
}
@@ -42,10 +44,12 @@ static inline struct ap_queue_status ap_tapq(ap_qid_t qid, unsigned long *info)
{
register unsigned long reg0 asm ("0") = qid;
register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm ("2") = 0UL;
+ register unsigned long reg2 asm ("2");
asm volatile(".long 0xb2af0000" /* PQAP(TAPQ) */
- : "+d" (reg0), "=d" (reg1), "+d" (reg2) : : "cc");
+ : "=d" (reg1), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
if (info)
*info = reg2;
return reg1;
@@ -59,13 +63,14 @@ static inline struct ap_queue_status ap_tapq(ap_qid_t qid, unsigned long *info)
*/
static inline struct ap_queue_status ap_rapq(ap_qid_t qid)
{
- register unsigned long reg0 asm ("0") = qid | 0x01000000UL;
+ register unsigned long reg0 asm ("0") = qid | (1UL << 24);
register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm ("2") = 0UL;
asm volatile(
".long 0xb2af0000" /* PQAP(RAPQ) */
- : "+d" (reg0), "=d" (reg1), "+d" (reg2) : : "cc");
+ : "=d" (reg1)
+ : "d" (reg0)
+ : "cc");
return reg1;
}
@@ -107,8 +112,8 @@ static inline struct ap_queue_status ap_aqic(ap_qid_t qid,
asm volatile(
".long 0xb2af0000" /* PQAP(AQIC) */
- : "+d" (reg0), "+d" (reg1_in), "=d" (reg1_out), "+d" (reg2)
- :
+ : "=d" (reg1_out)
+ : "d" (reg0), "d" (reg1_in), "d" (reg2)
: "cc");
return reg1_out;
}
@@ -120,17 +125,17 @@ static inline struct ap_queue_status ap_aqic(ap_qid_t qid,
*/
static inline int ap_qci(void *config)
{
- register unsigned long reg0 asm ("0") = 0x04000000UL;
+ register unsigned long reg0 asm ("0") = 4UL << 24;
register unsigned long reg1 asm ("1") = -EINVAL;
register void *reg2 asm ("2") = (void *) config;
asm volatile(
".long 0xb2af0000\n" /* PQAP(QCI) */
- "0: la %1,0\n"
+ "0: la %0,0\n"
"1:\n"
EX_TABLE(0b, 1b)
- : "+d" (reg0), "+d" (reg1), "+d" (reg2)
- :
+ : "+d" (reg1)
+ : "d" (reg0), "d" (reg2)
: "cc", "memory");
return reg1;
@@ -169,12 +174,13 @@ static inline struct ap_queue_status ap_qact(ap_qid_t qid, int ifbit,
| ((ifbit & 0x01) << 22);
register unsigned long reg1_in asm ("1") = apinfo->val;
register struct ap_queue_status reg1_out asm ("1");
- register unsigned long reg2 asm ("2") = 0;
+ register unsigned long reg2 asm ("2");
asm volatile(
".long 0xb2af0000" /* PQAP(QACT) */
- : "+d" (reg0), "+d" (reg1_in), "=d" (reg1_out), "+d" (reg2)
- : : "cc");
+ : "+d" (reg1_in), "=d" (reg1_out), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
apinfo->val = reg2;
return reg1_out;
}
--
1.7.1
From: Harald Freudenberger <[email protected]>
Move all the inline functions from the ap bus header
file ap_asm.h into the in-kernel api header file
arch/s390/include/asm/ap.h so that KVM can make use
of all the low level AP functions.
Signed-off-by: Harald Freudenberger <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
drivers/s390/crypto/ap_bus.c | 21 +---
drivers/s390/crypto/ap_bus.h | 1 +
drivers/s390/crypto/ap_card.c | 1 -
drivers/s390/crypto/ap_queue.c | 1 -
6 files changed, 259 insertions(+), 310 deletions(-)
delete mode 100644 drivers/s390/crypto/ap_asm.h
diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
index c1bedb4..046e044 100644
--- a/arch/s390/include/asm/ap.h
+++ b/arch/s390/include/asm/ap.h
@@ -47,6 +47,50 @@ struct ap_queue_status {
};
/**
+ * ap_intructions_available() - Test if AP instructions are available.
+ *
+ * Returns 0 if the AP instructions are installed.
+ */
+static inline int ap_instructions_available(void)
+{
+ register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
+ register unsigned long reg1 asm ("1") = -ENODEV;
+ register unsigned long reg2 asm ("2");
+
+ asm volatile(
+ " .long 0xb2af0000\n" /* PQAP(TAPQ) */
+ "0: la %0,0\n"
+ "1:\n"
+ EX_TABLE(0b, 1b)
+ : "+d" (reg1), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
+ * ap_tapq(): Test adjunct processor queue.
+ * @qid: The AP queue number
+ * @info: Pointer to queue descriptor
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_tapq(ap_qid_t qid, unsigned long *info)
+{
+ register unsigned long reg0 asm ("0") = qid;
+ register struct ap_queue_status reg1 asm ("1");
+ register unsigned long reg2 asm ("2");
+
+ asm volatile(".long 0xb2af0000" /* PQAP(TAPQ) */
+ : "=d" (reg1), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
+ if (info)
+ *info = reg2;
+ return reg1;
+}
+
+/**
* ap_test_queue(): Test adjunct processor queue.
* @qid: The AP queue number
* @tbit: Test facilities bit
@@ -54,10 +98,57 @@ struct ap_queue_status {
*
* Returns AP queue status structure.
*/
-struct ap_queue_status ap_test_queue(ap_qid_t qid,
- int tbit,
- unsigned long *info);
+static inline struct ap_queue_status ap_test_queue(ap_qid_t qid,
+ int tbit,
+ unsigned long *info)
+{
+ if (tbit)
+ qid |= 1UL << 23; /* set T bit*/
+ return ap_tapq(qid, info);
+}
+/**
+ * ap_pqap_rapq(): Reset adjunct processor queue.
+ * @qid: The AP queue number
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_rapq(ap_qid_t qid)
+{
+ register unsigned long reg0 asm ("0") = qid | (1UL << 24);
+ register struct ap_queue_status reg1 asm ("1");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(RAPQ) */
+ : "=d" (reg1)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
+ * ap_pqap_zapq(): Reset and zeroize adjunct processor queue.
+ * @qid: The AP queue number
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_zapq(ap_qid_t qid)
+{
+ register unsigned long reg0 asm ("0") = qid | (2UL << 24);
+ register struct ap_queue_status reg1 asm ("1");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(ZAPQ) */
+ : "=d" (reg1)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
+ * struct ap_config_info - convenience struct for AP crypto
+ * config info as returned by the ap_qci() function.
+ */
struct ap_config_info {
unsigned int apsc : 1; /* S bit */
unsigned int apxa : 1; /* N bit */
@@ -74,50 +165,189 @@ struct ap_config_info {
unsigned char _reserved4[16];
} __aligned(8);
-/*
- * ap_query_configuration(): Fetch cryptographic config info
+/**
+ * ap_qci(): Get AP configuration data
*
- * Returns the ap configuration info fetched via PQAP(QCI).
- * On success 0 is returned, on failure a negative errno
- * is returned, e.g. if the PQAP(QCI) instruction is not
- * available, the return value will be -EOPNOTSUPP.
+ * Returns 0 on success, or -EOPNOTSUPP.
*/
-int ap_query_configuration(struct ap_config_info *info);
+static inline int ap_qci(struct ap_config_info *config)
+{
+ register unsigned long reg0 asm ("0") = 4UL << 24;
+ register unsigned long reg1 asm ("1") = -EOPNOTSUPP;
+ register struct ap_config_info *reg2 asm ("2") = config;
+
+ asm volatile(
+ ".long 0xb2af0000\n" /* PQAP(QCI) */
+ "0: la %0,0\n"
+ "1:\n"
+ EX_TABLE(0b, 1b)
+ : "+d" (reg1)
+ : "d" (reg0), "d" (reg2)
+ : "cc", "memory");
+
+ return reg1;
+}
/*
* struct ap_qirq_ctrl - convenient struct for easy invocation
- * of the ap_queue_irq_ctrl() function. This struct is passed
- * as GR1 parameter to the PQAP(AQIC) instruction. For details
- * please see the AR documentation.
+ * of the ap_aqic() function. This struct is passed as GR1
+ * parameter to the PQAP(AQIC) instruction. For details please
+ * see the AR documentation.
*/
struct ap_qirq_ctrl {
unsigned int _res1 : 8;
- unsigned int zone : 8; /* zone info */
- unsigned int ir : 1; /* ir flag: enable (1) or disable (0) irq */
+ unsigned int zone : 8; /* zone info */
+ unsigned int ir : 1; /* ir flag: enable (1) or disable (0) irq */
unsigned int _res2 : 4;
- unsigned int gisc : 3; /* guest isc field */
+ unsigned int gisc : 3; /* guest isc field */
unsigned int _res3 : 6;
- unsigned int gf : 2; /* gisa format */
+ unsigned int gf : 2; /* gisa format */
unsigned int _res4 : 1;
- unsigned int gisa : 27; /* gisa origin */
+ unsigned int gisa : 27; /* gisa origin */
unsigned int _res5 : 1;
- unsigned int isc : 3; /* irq sub class */
+ unsigned int isc : 3; /* irq sub class */
};
/**
- * ap_queue_irq_ctrl(): Control interruption on a AP queue.
+ * ap_aqic(): Control interruption for a specific AP.
* @qid: The AP queue number
- * @qirqctrl: struct ap_qirq_ctrl, see above
+ * @qirqctrl: struct ap_qirq_ctrl (64 bit value)
* @ind: The notification indicator byte
*
* Returns AP queue status.
+ */
+static inline struct ap_queue_status ap_aqic(ap_qid_t qid,
+ struct ap_qirq_ctrl qirqctrl,
+ void *ind)
+{
+ register unsigned long reg0 asm ("0") = qid | (3UL << 24);
+ register struct ap_qirq_ctrl reg1_in asm ("1") = qirqctrl;
+ register struct ap_queue_status reg1_out asm ("1");
+ register void *reg2 asm ("2") = ind;
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(AQIC) */
+ : "=d" (reg1_out)
+ : "d" (reg0), "d" (reg1_in), "d" (reg2)
+ : "cc");
+ return reg1_out;
+}
+
+/*
+ * union ap_qact_ap_info - used together with the
+ * ap_aqic() function to provide a convenient way
+ * to handle the ap info needed by the qact function.
+ */
+union ap_qact_ap_info {
+ unsigned long val;
+ struct {
+ unsigned int : 3;
+ unsigned int mode : 3;
+ unsigned int : 26;
+ unsigned int cat : 8;
+ unsigned int : 8;
+ unsigned char ver[2];
+ };
+};
+
+/**
+ * ap_qact(): Query AP combatibility type.
+ * @qid: The AP queue number
+ * @apinfo: On input the info about the AP queue. On output the
+ * alternate AP queue info provided by the qact function
+ * in GR2 is stored in.
*
- * Control interruption on the given AP queue.
- * Just a simple wrapper function for the low level PQAP(AQIC)
- * instruction available for other kernel modules.
+ * Returns AP queue status. Check response_code field for failures.
*/
-struct ap_queue_status ap_queue_irq_ctrl(ap_qid_t qid,
- struct ap_qirq_ctrl qirqctrl,
- void *ind);
+static inline struct ap_queue_status ap_qact(ap_qid_t qid, int ifbit,
+ union ap_qact_ap_info *apinfo)
+{
+ register unsigned long reg0 asm ("0") = qid | (5UL << 24)
+ | ((ifbit & 0x01) << 22);
+ register unsigned long reg1_in asm ("1") = apinfo->val;
+ register struct ap_queue_status reg1_out asm ("1");
+ register unsigned long reg2 asm ("2");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(QACT) */
+ : "+d" (reg1_in), "=d" (reg1_out), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
+ apinfo->val = reg2;
+ return reg1_out;
+}
+
+/**
+ * ap_nqap(): Send message to adjunct processor queue.
+ * @qid: The AP queue number
+ * @psmid: The program supplied message identifier
+ * @msg: The message text
+ * @length: The message length
+ *
+ * Returns AP queue status structure.
+ * Condition code 1 on NQAP can't happen because the L bit is 1.
+ * Condition code 2 on NQAP also means the send is incomplete,
+ * because a segment boundary was reached. The NQAP is repeated.
+ */
+static inline struct ap_queue_status ap_nqap(ap_qid_t qid,
+ unsigned long long psmid,
+ void *msg, size_t length)
+{
+ register unsigned long reg0 asm ("0") = qid | 0x40000000UL;
+ register struct ap_queue_status reg1 asm ("1");
+ register unsigned long reg2 asm ("2") = (unsigned long) msg;
+ register unsigned long reg3 asm ("3") = (unsigned long) length;
+ register unsigned long reg4 asm ("4") = (unsigned int) (psmid >> 32);
+ register unsigned long reg5 asm ("5") = psmid & 0xffffffff;
+
+ asm volatile (
+ "0: .long 0xb2ad0042\n" /* NQAP */
+ " brc 2,0b"
+ : "+d" (reg0), "=d" (reg1), "+d" (reg2), "+d" (reg3)
+ : "d" (reg4), "d" (reg5)
+ : "cc", "memory");
+ return reg1;
+}
+
+/**
+ * ap_dqap(): Receive message from adjunct processor queue.
+ * @qid: The AP queue number
+ * @psmid: Pointer to program supplied message identifier
+ * @msg: The message text
+ * @length: The message length
+ *
+ * Returns AP queue status structure.
+ * Condition code 1 on DQAP means the receive has taken place
+ * but only partially. The response is incomplete, hence the
+ * DQAP is repeated.
+ * Condition code 2 on DQAP also means the receive is incomplete,
+ * this time because a segment boundary was reached. Again, the
+ * DQAP is repeated.
+ * Note that gpr2 is used by the DQAP instruction to keep track of
+ * any 'residual' length, in case the instruction gets interrupted.
+ * Hence it gets zeroed before the instruction.
+ */
+static inline struct ap_queue_status ap_dqap(ap_qid_t qid,
+ unsigned long long *psmid,
+ void *msg, size_t length)
+{
+ register unsigned long reg0 asm("0") = qid | 0x80000000UL;
+ register struct ap_queue_status reg1 asm ("1");
+ register unsigned long reg2 asm("2") = 0UL;
+ register unsigned long reg4 asm("4") = (unsigned long) msg;
+ register unsigned long reg5 asm("5") = (unsigned long) length;
+ register unsigned long reg6 asm("6") = 0UL;
+ register unsigned long reg7 asm("7") = 0UL;
+
+
+ asm volatile(
+ "0: .long 0xb2ae0064\n" /* DQAP */
+ " brc 6,0b\n"
+ : "+d" (reg0), "=d" (reg1), "+d" (reg2),
+ "+d" (reg4), "+d" (reg5), "+d" (reg6), "+d" (reg7)
+ : : "cc", "memory");
+ *psmid = (((unsigned long long) reg6) << 32) + reg7;
+ return reg1;
+}
#endif /* _ASM_S390_AP_H_ */
diff --git a/drivers/s390/crypto/ap_asm.h b/drivers/s390/crypto/ap_asm.h
deleted file mode 100644
index e22ee12..0000000
--- a/drivers/s390/crypto/ap_asm.h
+++ /dev/null
@@ -1,261 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Copyright IBM Corp. 2016
- * Author(s): Martin Schwidefsky <[email protected]>
- *
- * Adjunct processor bus inline assemblies.
- */
-
-#ifndef _AP_ASM_H_
-#define _AP_ASM_H_
-
-#include <asm/isc.h>
-
-/**
- * ap_intructions_available() - Test if AP instructions are available.
- *
- * Returns 0 if the AP instructions are installed.
- */
-static inline int ap_instructions_available(void)
-{
- register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
- register unsigned long reg1 asm ("1") = -ENODEV;
- register unsigned long reg2 asm ("2");
-
- asm volatile(
- " .long 0xb2af0000\n" /* PQAP(TAPQ) */
- "0: la %0,0\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+d" (reg1), "=d" (reg2)
- : "d" (reg0)
- : "cc");
- return reg1;
-}
-
-/**
- * ap_tapq(): Test adjunct processor queue.
- * @qid: The AP queue number
- * @info: Pointer to queue descriptor
- *
- * Returns AP queue status structure.
- */
-static inline struct ap_queue_status ap_tapq(ap_qid_t qid, unsigned long *info)
-{
- register unsigned long reg0 asm ("0") = qid;
- register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm ("2");
-
- asm volatile(".long 0xb2af0000" /* PQAP(TAPQ) */
- : "=d" (reg1), "=d" (reg2)
- : "d" (reg0)
- : "cc");
- if (info)
- *info = reg2;
- return reg1;
-}
-
-/**
- * ap_pqap_rapq(): Reset adjunct processor queue.
- * @qid: The AP queue number
- *
- * Returns AP queue status structure.
- */
-static inline struct ap_queue_status ap_rapq(ap_qid_t qid)
-{
- register unsigned long reg0 asm ("0") = qid | (1UL << 24);
- register struct ap_queue_status reg1 asm ("1");
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(RAPQ) */
- : "=d" (reg1)
- : "d" (reg0)
- : "cc");
- return reg1;
-}
-
-/**
- * ap_pqap_zapq(): Reset and zeroize adjunct processor queue.
- * @qid: The AP queue number
- *
- * Returns AP queue status structure.
- */
-static inline struct ap_queue_status ap_zapq(ap_qid_t qid)
-{
- register unsigned long reg0 asm ("0") = qid | (2UL << 24);
- register struct ap_queue_status reg1 asm ("1");
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(ZAPQ) */
- : "=d" (reg1)
- : "d" (reg0)
- : "cc");
- return reg1;
-}
-
-/**
- * ap_aqic(): Control interruption for a specific AP.
- * @qid: The AP queue number
- * @qirqctrl: struct ap_qirq_ctrl (64 bit value)
- * @ind: The notification indicator byte
- *
- * Returns AP queue status.
- */
-static inline struct ap_queue_status ap_aqic(ap_qid_t qid,
- struct ap_qirq_ctrl qirqctrl,
- void *ind)
-{
- register unsigned long reg0 asm ("0") = qid | (3UL << 24);
- register struct ap_qirq_ctrl reg1_in asm ("1") = qirqctrl;
- register struct ap_queue_status reg1_out asm ("1");
- register void *reg2 asm ("2") = ind;
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(AQIC) */
- : "=d" (reg1_out)
- : "d" (reg0), "d" (reg1_in), "d" (reg2)
- : "cc");
- return reg1_out;
-}
-
-/**
- * ap_qci(): Get AP configuration data
- *
- * Returns 0 on success, or -EOPNOTSUPP.
- */
-static inline int ap_qci(void *config)
-{
- register unsigned long reg0 asm ("0") = 4UL << 24;
- register unsigned long reg1 asm ("1") = -EINVAL;
- register void *reg2 asm ("2") = (void *) config;
-
- asm volatile(
- ".long 0xb2af0000\n" /* PQAP(QCI) */
- "0: la %0,0\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+d" (reg1)
- : "d" (reg0), "d" (reg2)
- : "cc", "memory");
-
- return reg1;
-}
-
-/*
- * union ap_qact_ap_info - used together with the
- * ap_aqic() function to provide a convenient way
- * to handle the ap info needed by the qact function.
- */
-union ap_qact_ap_info {
- unsigned long val;
- struct {
- unsigned int : 3;
- unsigned int mode : 3;
- unsigned int : 26;
- unsigned int cat : 8;
- unsigned int : 8;
- unsigned char ver[2];
- };
-};
-
-/**
- * ap_qact(): Query AP combatibility type.
- * @qid: The AP queue number
- * @apinfo: On input the info about the AP queue. On output the
- * alternate AP queue info provided by the qact function
- * in GR2 is stored in.
- *
- * Returns AP queue status. Check response_code field for failures.
- */
-static inline struct ap_queue_status ap_qact(ap_qid_t qid, int ifbit,
- union ap_qact_ap_info *apinfo)
-{
- register unsigned long reg0 asm ("0") = qid | (5UL << 24)
- | ((ifbit & 0x01) << 22);
- register unsigned long reg1_in asm ("1") = apinfo->val;
- register struct ap_queue_status reg1_out asm ("1");
- register unsigned long reg2 asm ("2");
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(QACT) */
- : "+d" (reg1_in), "=d" (reg1_out), "=d" (reg2)
- : "d" (reg0)
- : "cc");
- apinfo->val = reg2;
- return reg1_out;
-}
-
-/**
- * ap_nqap(): Send message to adjunct processor queue.
- * @qid: The AP queue number
- * @psmid: The program supplied message identifier
- * @msg: The message text
- * @length: The message length
- *
- * Returns AP queue status structure.
- * Condition code 1 on NQAP can't happen because the L bit is 1.
- * Condition code 2 on NQAP also means the send is incomplete,
- * because a segment boundary was reached. The NQAP is repeated.
- */
-static inline struct ap_queue_status ap_nqap(ap_qid_t qid,
- unsigned long long psmid,
- void *msg, size_t length)
-{
- register unsigned long reg0 asm ("0") = qid | 0x40000000UL;
- register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm ("2") = (unsigned long) msg;
- register unsigned long reg3 asm ("3") = (unsigned long) length;
- register unsigned long reg4 asm ("4") = (unsigned int) (psmid >> 32);
- register unsigned long reg5 asm ("5") = psmid & 0xffffffff;
-
- asm volatile (
- "0: .long 0xb2ad0042\n" /* NQAP */
- " brc 2,0b"
- : "+d" (reg0), "=d" (reg1), "+d" (reg2), "+d" (reg3)
- : "d" (reg4), "d" (reg5)
- : "cc", "memory");
- return reg1;
-}
-
-/**
- * ap_dqap(): Receive message from adjunct processor queue.
- * @qid: The AP queue number
- * @psmid: Pointer to program supplied message identifier
- * @msg: The message text
- * @length: The message length
- *
- * Returns AP queue status structure.
- * Condition code 1 on DQAP means the receive has taken place
- * but only partially. The response is incomplete, hence the
- * DQAP is repeated.
- * Condition code 2 on DQAP also means the receive is incomplete,
- * this time because a segment boundary was reached. Again, the
- * DQAP is repeated.
- * Note that gpr2 is used by the DQAP instruction to keep track of
- * any 'residual' length, in case the instruction gets interrupted.
- * Hence it gets zeroed before the instruction.
- */
-static inline struct ap_queue_status ap_dqap(ap_qid_t qid,
- unsigned long long *psmid,
- void *msg, size_t length)
-{
- register unsigned long reg0 asm("0") = qid | 0x80000000UL;
- register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm("2") = 0UL;
- register unsigned long reg4 asm("4") = (unsigned long) msg;
- register unsigned long reg5 asm("5") = (unsigned long) length;
- register unsigned long reg6 asm("6") = 0UL;
- register unsigned long reg7 asm("7") = 0UL;
-
-
- asm volatile(
- "0: .long 0xb2ae0064\n" /* DQAP */
- " brc 6,0b\n"
- : "+d" (reg0), "=d" (reg1), "+d" (reg2),
- "+d" (reg4), "+d" (reg5), "+d" (reg6), "+d" (reg7)
- : : "cc", "memory");
- *psmid = (((unsigned long long) reg6) << 32) + reg7;
- return reg1;
-}
-
-#endif /* _AP_ASM_H_ */
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index 35a0c2b..c0a6723 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -36,7 +36,6 @@
#include <linux/debugfs.h>
#include "ap_bus.h"
-#include "ap_asm.h"
#include "ap_debug.h"
/*
@@ -174,24 +173,6 @@ static inline int ap_qact_available(void)
return 0;
}
-/**
- * ap_test_queue(): Test adjunct processor queue.
- * @qid: The AP queue number
- * @tbit: Test facilities bit
- * @info: Pointer to queue descriptor
- *
- * Returns AP queue status structure.
- */
-struct ap_queue_status ap_test_queue(ap_qid_t qid,
- int tbit,
- unsigned long *info)
-{
- if (tbit)
- qid |= 1UL << 23; /* set T bit*/
- return ap_tapq(qid, info);
-}
-EXPORT_SYMBOL(ap_test_queue);
-
/*
* ap_query_configuration(): Fetch cryptographic config info
*
@@ -200,7 +181,7 @@ struct ap_queue_status ap_test_queue(ap_qid_t qid,
* is returned, e.g. if the PQAP(QCI) instruction is not
* available, the return value will be -EOPNOTSUPP.
*/
-int ap_query_configuration(struct ap_config_info *info)
+static inline int ap_query_configuration(struct ap_config_info *info)
{
if (!ap_configuration_available())
return -EOPNOTSUPP;
diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h
index 6a273c5..9365419 100644
--- a/drivers/s390/crypto/ap_bus.h
+++ b/drivers/s390/crypto/ap_bus.h
@@ -15,6 +15,7 @@
#include <linux/device.h>
#include <linux/types.h>
+#include <asm/isc.h>
#include <asm/ap.h>
#define AP_DEVICES 256 /* Number of AP devices. */
diff --git a/drivers/s390/crypto/ap_card.c b/drivers/s390/crypto/ap_card.c
index 2c726df..c13e432 100644
--- a/drivers/s390/crypto/ap_card.c
+++ b/drivers/s390/crypto/ap_card.c
@@ -14,7 +14,6 @@
#include <asm/facility.h>
#include "ap_bus.h"
-#include "ap_asm.h"
/*
* AP card related attributes.
diff --git a/drivers/s390/crypto/ap_queue.c b/drivers/s390/crypto/ap_queue.c
index ba3a2e1..d83c1fa 100644
--- a/drivers/s390/crypto/ap_queue.c
+++ b/drivers/s390/crypto/ap_queue.c
@@ -14,7 +14,6 @@
#include <asm/facility.h>
#include "ap_bus.h"
-#include "ap_asm.h"
/**
* ap_queue_irq_ctrl(): Control interruption on a AP queue.
--
1.7.1
From: Harald Freudenberger <[email protected]>
Added new inline function ap_pqap_zapq()
which is a C inline function wrapper for
the AP PQAP(ZAPQ) instruction.
Signed-off-by: Harald Freudenberger <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
Reviewed-by: Tony Krowiak <[email protected]>
---
drivers/s390/crypto/ap_asm.h | 19 +++++++++++++++++++
1 files changed, 19 insertions(+), 0 deletions(-)
diff --git a/drivers/s390/crypto/ap_asm.h b/drivers/s390/crypto/ap_asm.h
index 16b59ce..b22d30a 100644
--- a/drivers/s390/crypto/ap_asm.h
+++ b/drivers/s390/crypto/ap_asm.h
@@ -70,6 +70,25 @@ static inline struct ap_queue_status ap_rapq(ap_qid_t qid)
}
/**
+ * ap_pqap_zapq(): Reset and zeroize adjunct processor queue.
+ * @qid: The AP queue number
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_zapq(ap_qid_t qid)
+{
+ register unsigned long reg0 asm ("0") = qid | (2UL << 24);
+ register struct ap_queue_status reg1 asm ("1");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(ZAPQ) */
+ : "=d" (reg1)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
* ap_aqic(): Control interruption for a specific AP.
* @qid: The AP queue number
* @qirqctrl: struct ap_qirq_ctrl (64 bit value)
--
1.7.1
From: Tony Krowiak <[email protected]>
This patch refactors the code that initializes and sets up the
crypto configuration for a guest. The following changes are
implemented via this patch:
1. Prior to the introduction of AP device virtualization, it
was not necessary to provide guest access to the CRYCB
unless the MSA extension 3 (MSAX3) facility was installed
on the host system. With the introduction of AP device
virtualization, the CRYCB must be made accessible to the
guest if the AP instructions are installed on the host
and are to be provided to the guest.
2. Introduces a flag indicating AP instructions executed on
the guest shall be interpreted by the firmware. It is
initialized to indicate AP instructions are to be
to be interpreted and is used to set the SIE bit for
each vcpu during vcpu setup.
Signed-off-by: Tony Krowiak <[email protected]>
Reviewed-by: Halil Pasic <[email protected]>
Acked-by: Christian Borntraeger <[email protected]>
Tested-by: Michael Mueller <[email protected]>
Tested-by: Farhan Ali <[email protected]>
Signed-off-by: Christian Borntraeger <[email protected]>
---
arch/s390/include/asm/kvm_host.h | 3 +
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/kvm-s390.c | 86 ++++++++++++++++++++------------------
3 files changed, 49 insertions(+), 41 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index af39561..0c13f61 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -187,6 +187,7 @@ struct kvm_s390_sie_block {
#define ECA_AIV 0x00200000
#define ECA_VX 0x00020000
#define ECA_PROTEXCI 0x00002000
+#define ECA_APIE 0x00000008
#define ECA_SII 0x00000001
__u32 eca; /* 0x004c */
#define ICPT_INST 0x04
@@ -256,6 +257,7 @@ struct kvm_s390_sie_block {
__u8 reservede4[4]; /* 0x00e4 */
__u64 tecmc; /* 0x00e8 */
__u8 reservedf0[12]; /* 0x00f0 */
+#define CRYCB_FORMAT_MASK 0x00000003
#define CRYCB_FORMAT1 0x00000001
#define CRYCB_FORMAT2 0x00000003
__u32 crycbd; /* 0x00fc */
@@ -714,6 +716,7 @@ struct kvm_s390_crypto {
__u32 crycbd;
__u8 aes_kw;
__u8 dea_kw;
+ __u8 apie;
};
#define APCB0_MASK_SIZE 1
diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index 4cdaa55..a580dec 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
#define KVM_S390_VM_CPU_FEAT_KSS 13
+#define KVM_S390_VM_CPU_FEAT_AP 14
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 19f4f44..f96a443 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -40,6 +40,7 @@
#include <asm/sclp.h>
#include <asm/cpacf.h>
#include <asm/timex.h>
+#include <asm/ap.h>
#include "kvm-s390.h"
#include "gaccess.h"
@@ -1881,49 +1882,37 @@ long kvm_arch_vm_ioctl(struct file *filp,
return r;
}
-static int kvm_s390_query_ap_config(u8 *config)
-{
- u32 fcn_code = 0x04000000UL;
- u32 cc = 0;
-
- memset(config, 0, 128);
- asm volatile(
- "lgr 0,%1\n"
- "lgr 2,%2\n"
- ".long 0xb2af0000\n" /* PQAP(QCI) */
- "0: ipm %0\n"
- "srl %0,28\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+r" (cc)
- : "r" (fcn_code), "r" (config)
- : "cc", "0", "2", "memory"
- );
-
- return cc;
-}
-
static int kvm_s390_apxa_installed(void)
{
- u8 config[128];
- int cc;
+ struct ap_config_info info;
- if (test_facility(12)) {
- cc = kvm_s390_query_ap_config(config);
-
- if (cc)
- pr_err("PQAP(QCI) failed with cc=%d", cc);
- else
- return config[0] & 0x40;
+ if (ap_instructions_available() == 0) {
+ if (ap_qci(&info) == 0)
+ return info.apxa;
}
return 0;
}
+/*
+ * The format of the crypto control block (CRYCB) is specified in the 3 low
+ * order bits of the CRYCB designation (CRYCBD) field as follows:
+ * Format 0: Neither the message security assist extension 3 (MSAX3) nor the
+ * AP extended addressing (APXA) facility are installed.
+ * Format 1: The APXA facility is not installed but the MSAX3 facility is.
+ * Format 2: Both the APXA and MSAX3 facilities are installed
+ */
static void kvm_s390_set_crycb_format(struct kvm *kvm)
{
kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
+ /* Clear the CRYCB format bits - i.e., set format 0 by default */
+ kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT_MASK);
+
+ /* Check whether MSAX3 is installed */
+ if (!test_kvm_facility(kvm, 76))
+ return;
+
if (kvm_s390_apxa_installed())
kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
else
@@ -1941,11 +1930,13 @@ static u64 kvm_s390_get_initial_cpuid(void)
static void kvm_s390_crypto_init(struct kvm *kvm)
{
- if (!test_kvm_facility(kvm, 76))
- return;
-
kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
kvm_s390_set_crycb_format(kvm);
+ /* Default setting indicating SIE shall interpret AP instructions */
+ kvm->arch.crypto.apie = 1;
+
+ if (!test_kvm_facility(kvm, 76))
+ return;
/* Enable AES/DEA protected key functions by default */
kvm->arch.crypto.aes_kw = 1;
@@ -2474,17 +2465,30 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
{
- if (!test_kvm_facility(vcpu->kvm, 76))
+ /*
+ * If neither the AP instructions nor the MSAX3 facility are installed
+ * on the host, then there is no need for a CRYCB in SIE because the
+ * they will not be installed on the guest either.
+ */
+ if (ap_instructions_available() && !test_facility(76))
return;
- vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
+ vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
+
+ vcpu->arch.sie_block->eca &= ~ECA_APIE;
+ if (vcpu->kvm->arch.crypto.apie &&
+ test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
+ vcpu->arch.sie_block->eca |= ECA_APIE;
- if (vcpu->kvm->arch.crypto.aes_kw)
- vcpu->arch.sie_block->ecb3 |= ECB3_AES;
- if (vcpu->kvm->arch.crypto.dea_kw)
- vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
+ /* If MSAX3 is installed on the guest, set up protected key support */
+ if (test_kvm_facility(vcpu->kvm, 76)) {
+ vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
- vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
+ if (vcpu->kvm->arch.crypto.aes_kw)
+ vcpu->arch.sie_block->ecb3 |= ECB3_AES;
+ if (vcpu->kvm->arch.crypto.dea_kw)
+ vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
+ }
}
void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
--
1.7.1
On 08.08.2018 16:44, Tony Krowiak wrote:
> From: Tony Krowiak <[email protected]>
The patches are also available on git.kernel.org:
https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git/log/?h=ap
On Wed, 8 Aug 2018 10:44:10 -0400
Tony Krowiak <[email protected]> wrote:
> From: Tony Krowiak <[email protected]>
>
> Several major objections were raised to design changes introduced in the v7
> patch series, so in order to avoid an extended discussion around these
> objections and to expedite acceptance of the series, the following changes
> have been made for v8:
>
> 1. Removed the AP bus's ability to designate queues as 'used by host' or as
> 'used by alternate driver(s)'. The bind/unbind sysfs interfaces will be
> used for managing the connection of AP queue devices to a zcrypt driver
> or the VFIO AP driver.
I don't think the idea of pools is bad per se; I mainly did not like
the sysfs interface and the dynamic interactions.
We can probably reintroduce something like that later on, if it is
still useful.
>
> 2. Removed the 'activate' sysfs interfaces which allowed for over
> provisioning of the mediated device as well as creation of mdevs with
> overlapping matrixes. It was pointed out that both of these enhancements
> break the mdev model. Consistency checking of the mdev matrix has
> therefore been returned to the mediated matrix device's sysfs interfaces
> for assigning adapters and domains:
>
> * Verify that APQNs assigned to the mediated device are bound to the
> VFIO AP device driver
>
> * Verify that no APQN assigned to the mediated matrix device is assigned
> to any other mediated matrix device.
Ok, that makes sense.
Where's point 3? :)
>
> 4. Reworked the handling of the CRYCB in vSIE based upon patches introduced
> by David in the mainline.
I had reviewed David's patches and they looked good to me.
>
> Notes:
> =====
>
> Patches 1-4 (by Harald) posted with this series are forthcoming via
> Martins tree and are based on changes in the ap driver/bus that we use as a
> foundation. They have been included here because some of the functions
> in this patch series are dependent upon them.
I don't remember anything contentious in these.
>
> Patches 5-6 (by David) are posted with this series because they are not
> currently in our master branch. Patches 19 and 20 of this series are
> dependent upon them. I believe David's patches are available in the
> mainline now.
I don't see them queued yet, but as said, they looked fine to me.
>
> This patch series works with the v6 QEMU patches. There is no new QEMU
> patchset version yet because there have been no review comments worthy of
> creating a new series; only a couple of extremely minor nits.
Once the kernel part is merged, I'd need a respin anyway due to the
kernel headers updates.
On 08/08/2018 12:25 PM, Cornelia Huck wrote:
> On Wed, 8 Aug 2018 10:44:10 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> From: Tony Krowiak <[email protected]>
>>
>> Several major objections were raised to design changes introduced in the v7
>> patch series, so in order to avoid an extended discussion around these
>> objections and to expedite acceptance of the series, the following changes
>> have been made for v8:
>>
>> 1. Removed the AP bus's ability to designate queues as 'used by host' or as
>> 'used by alternate driver(s)'. The bind/unbind sysfs interfaces will be
>> used for managing the connection of AP queue devices to a zcrypt driver
>> or the VFIO AP driver.
> I don't think the idea of pools is bad per se; I mainly did not like
> the sysfs interface and the dynamic interactions.
>
> We can probably reintroduce something like that later on, if it is
> still useful.
That may very well be the case, but we decided on bare bones expedite
acceptance.
>
>> 2. Removed the 'activate' sysfs interfaces which allowed for over
>> provisioning of the mediated device as well as creation of mdevs with
>> overlapping matrixes. It was pointed out that both of these enhancements
>> break the mdev model. Consistency checking of the mdev matrix has
>> therefore been returned to the mediated matrix device's sysfs interfaces
>> for assigning adapters and domains:
>>
>> * Verify that APQNs assigned to the mediated device are bound to the
>> VFIO AP device driver
>>
>> * Verify that no APQN assigned to the mediated matrix device is assigned
>> to any other mediated matrix device.
> Ok, that makes sense.
>
> Where's point 3? :)
That is the invisible point. Only the all-knowing, all-seeing can
discern its
presence ;)
>
>> 4. Reworked the handling of the CRYCB in vSIE based upon patches introduced
>> by David in the mainline.
> I had reviewed David's patches and they looked good to me.
Excellent!
>
>> Notes:
>> =====
>>
>> Patches 1-4 (by Harald) posted with this series are forthcoming via
>> Martins tree and are based on changes in the ap driver/bus that we use as a
>> foundation. They have been included here because some of the functions
>> in this patch series are dependent upon them.
> I don't remember anything contentious in these.
There weren't any issues. They are included here solely because they are
needed to
build the kernel and are not yet available in our master branch.
>
>> Patches 5-6 (by David) are posted with this series because they are not
>> currently in our master branch. Patches 19 and 20 of this series are
>> dependent upon them. I believe David's patches are available in the
>> mainline now.
> I don't see them queued yet, but as said, they looked fine to me.
I was told they are available in the mainline, but I'm not entirely sure
what that means
and I didn't verify it. They are included here precisely
because they are not yet available in our master branch and our code is
dependent upon
them.
>
>> This patch series works with the v6 QEMU patches. There is no new QEMU
>> patchset version yet because there have been no review comments worthy of
>> creating a new series; only a couple of extremely minor nits.
> Once the kernel part is merged, I'd need a respin anyway due to the
> kernel headers updates.
Got it.
>
On 08.08.2018 16:44, Tony Krowiak wrote:
> From: Tony Krowiak <[email protected]>
>
> This patch refactors the code that initializes and sets up the
> crypto configuration for a guest. The following changes are
> implemented via this patch:
>
> 1. Prior to the introduction of AP device virtualization, it
> was not necessary to provide guest access to the CRYCB
> unless the MSA extension 3 (MSAX3) facility was installed
> on the host system. With the introduction of AP device
> virtualization, the CRYCB must be made accessible to the
> guest if the AP instructions are installed on the host
> and are to be provided to the guest.
>
> 2. Introduces a flag indicating AP instructions executed on
> the guest shall be interpreted by the firmware. It is
> initialized to indicate AP instructions are to be
> to be interpreted and is used to set the SIE bit for
> each vcpu during vcpu setup.
>
> Signed-off-by: Tony Krowiak <[email protected]>
> Reviewed-by: Halil Pasic <[email protected]>
> Acked-by: Christian Borntraeger <[email protected]>
> Tested-by: Michael Mueller <[email protected]>
> Tested-by: Farhan Ali <[email protected]>
> Signed-off-by: Christian Borntraeger <[email protected]>
Acked-by: Janosch Frank <[email protected]>
> ---
> arch/s390/include/asm/kvm_host.h | 3 +
> arch/s390/include/uapi/asm/kvm.h | 1 +
> arch/s390/kvm/kvm-s390.c | 86 ++++++++++++++++++++------------------
> 3 files changed, 49 insertions(+), 41 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index af39561..0c13f61 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -187,6 +187,7 @@ struct kvm_s390_sie_block {
> #define ECA_AIV 0x00200000
> #define ECA_VX 0x00020000
> #define ECA_PROTEXCI 0x00002000
> +#define ECA_APIE 0x00000008
> #define ECA_SII 0x00000001
> __u32 eca; /* 0x004c */
> #define ICPT_INST 0x04
> @@ -256,6 +257,7 @@ struct kvm_s390_sie_block {
> __u8 reservede4[4]; /* 0x00e4 */
> __u64 tecmc; /* 0x00e8 */
> __u8 reservedf0[12]; /* 0x00f0 */
> +#define CRYCB_FORMAT_MASK 0x00000003
> #define CRYCB_FORMAT1 0x00000001
> #define CRYCB_FORMAT2 0x00000003
> __u32 crycbd; /* 0x00fc */
> @@ -714,6 +716,7 @@ struct kvm_s390_crypto {
> __u32 crycbd;
> __u8 aes_kw;
> __u8 dea_kw;
> + __u8 apie;
In the last review I wanted a comment here to know what they do.
> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
> {
> - if (!test_kvm_facility(vcpu->kvm, 76))
> + /*
> + * If neither the AP instructions nor the MSAX3 facility are installed
> + * on the host, then there is no need for a CRYCB in SIE because the
> + * they will not be installed on the guest either.
the they
> + */
> + if (ap_instructions_available() && !test_facility(76))
> return;
I know you're not responsible for that one :) but 0 being the wanted
value here is a bit counter-intuitive.
>
> - vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
> + vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> +
> + vcpu->arch.sie_block->eca &= ~ECA_APIE;
The scb is zero allocated, are the ECA and the ECB3s set somewhere
in-between, or is that your way of making sure the controls are
definitely gone for good?
> + if (vcpu->kvm->arch.crypto.apie &&
> + test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
> + vcpu->arch.sie_block->eca |= ECA_APIE;
>
> - if (vcpu->kvm->arch.crypto.aes_kw)
> - vcpu->arch.sie_block->ecb3 |= ECB3_AES;
> - if (vcpu->kvm->arch.crypto.dea_kw)
> - vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
> + /* If MSAX3 is installed on the guest, set up protected key support */
> + if (test_kvm_facility(vcpu->kvm, 76)) {
> + vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>
> - vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> + if (vcpu->kvm->arch.crypto.aes_kw)
> + vcpu->arch.sie_block->ecb3 |= ECB3_AES;
> + if (vcpu->kvm->arch.crypto.dea_kw)
> + vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
> + }
> }
>
> void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
>
On 08.08.2018 16:44, Tony Krowiak wrote:
> From: Pierre Morel <[email protected]>
> +#define ECA_APIE 0x00000008
That shouldn't be necessary, it's defined in kvm_host.h which vsie.c
includes. Or is it not?
> +static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> +{
> + struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
> + struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
> + const uint32_t crycbd_o = READ_ONCE(scb_o->crycbd);
> + const u32 crycb_addr = crycbd_o & 0x7ffffff8U;
> + int g2_fmt = vcpu->arch.sie_block->crycbd & CRYCB_FORMAT_MASK;
> + int g3_fmt = crycbd_o & CRYCB_FORMAT_MASK;
> + int g2_apie, g2_msa3, g3_apie, g3_msa3;
> + int size, ret;
> +
> + /* crycb should not cross a page boundary */
> + size = (g3_fmt == CRYCB_FORMAT2) ? 0x100 : 0x80;
> + if ((crycb_addr & PAGE_MASK) != ((crycb_addr + size) & PAGE_MASK))
> + return set_validity_icpt(scb_s, 0x003CU);
> +
> + g2_apie = vcpu->arch.sie_block->eca & ECA_APIE;
> + g3_apie = scb_o->eca & g2_apie;
> +
> + g2_msa3 = test_kvm_facility(vcpu->kvm, 76);
> + g3_msa3 = (g3_fmt != CRYCB_FORMAT0) & g2_msa3;
> +
> + scb_s->crycbd = 0;
> + /* If no AP instructions and no keys we just set crycbd to 0 */
> + if (!(g3_apie || g3_msa3))
> + return 0;
> +
> + if (!crycb_addr)
> + return set_validity_icpt(scb_s, 0x0039U);
> +
> + if (g3_apie) {
> + ret = copy_apcb(vcpu, vsie_page, g2_fmt, g3_fmt);
> + if (ret)
> + goto out;
s/goto out;/return ret;/
> + scb_s->eca |= g3_apie;
> + }
> +
> + if (g3_msa3)
> + ret = copy_key_masks(vcpu, vsie_page);
> +
> + if (!ret)
> + scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) | g2_fmt;
> +
> +out:
> + return ret;
> +}
> +
> /* shadow (round up/down) the ibc to avoid validity icpt */
> static void prepare_ibc(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
> {
>
On 09/08/2018 08:20, Janosch Frank wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Pierre Morel <[email protected]>
>> +#define ECA_APIE 0x00000008
> That shouldn't be necessary, it's defined in kvm_host.h which vsie.c
> includes. Or is it not?
This was forgotten here for a long long time!
You are right I remove it.
>
>> +static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
>> +{
>> + struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
>> + struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
>> + const uint32_t crycbd_o = READ_ONCE(scb_o->crycbd);
>> + const u32 crycb_addr = crycbd_o & 0x7ffffff8U;
>> + int g2_fmt = vcpu->arch.sie_block->crycbd & CRYCB_FORMAT_MASK;
>> + int g3_fmt = crycbd_o & CRYCB_FORMAT_MASK;
>> + int g2_apie, g2_msa3, g3_apie, g3_msa3;
>> + int size, ret;
>> +
>> + /* crycb should not cross a page boundary */
>> + size = (g3_fmt == CRYCB_FORMAT2) ? 0x100 : 0x80;
>> + if ((crycb_addr & PAGE_MASK) != ((crycb_addr + size) & PAGE_MASK))
>> + return set_validity_icpt(scb_s, 0x003CU);
>> +
>> + g2_apie = vcpu->arch.sie_block->eca & ECA_APIE;
>> + g3_apie = scb_o->eca & g2_apie;
>> +
>> + g2_msa3 = test_kvm_facility(vcpu->kvm, 76);
>> + g3_msa3 = (g3_fmt != CRYCB_FORMAT0) & g2_msa3;
>> +
>> + scb_s->crycbd = 0;
>> + /* If no AP instructions and no keys we just set crycbd to 0 */
>> + if (!(g3_apie || g3_msa3))
>> + return 0;
>> +
>> + if (!crycb_addr)
>> + return set_validity_icpt(scb_s, 0x0039U);
>> +
>> + if (g3_apie) {
>> + ret = copy_apcb(vcpu, vsie_page, g2_fmt, g3_fmt);
>> + if (ret)
>> + goto out;
> s/goto out;/return ret;/
better
Thanks
Pierre
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
On 08.08.2018 16:44, Tony Krowiak wrote:
> From: Pierre Morel <[email protected]>
>
> When we clear the Crypto Control Block (CRYCB) used by a guest
> level 2, the vSIE shadow CRYCB for guest level 3 must be updated
> before the guest uses it.
>
> We achieve this by using the KVM_REQ_VSIE_RESTART synchronous
> request for each vCPU belonging to the guest to force the reload
> of the shadow CRYCB before rerunning the guest level 3.
>
> Signed-off-by: Pierre Morel <[email protected]>
> ---
> arch/s390/kvm/kvm-s390.c | 7 +++++++
> 1 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 0d03249..9203f0b 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -1921,6 +1921,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>
> void kvm_arch_crypto_clear_masks(struct kvm *kvm)
> {
> + int i;
> + struct kvm_vcpu *vcpu;
> +
> mutex_lock(&kvm->lock);
> kvm_s390_vcpu_block_all(kvm);
>
> @@ -1929,6 +1932,10 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm)
> memset(&kvm->arch.crypto.crycb->apcb1, 0,
> sizeof(kvm->arch.crypto.crycb->apcb1));
>
> + /* recreate the shadow crycb for each vcpu */
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
We have kvm_s390_sync_request_broadcast(), which should boil down to a
one-liner
> +
> kvm_s390_vcpu_unblock_all(kvm);
> mutex_unlock(&kvm->lock);
> }
>
--
Thanks,
David / dhildenb
On 08.08.2018 16:44, Tony Krowiak wrote:
> From: Tony Krowiak <[email protected]>
>
> Introduces a new CPU model feature and two CPU model
> facilities to support AP virtualization for KVM guests.
>
> CPU model feature:
>
> The KVM_S390_VM_CPU_FEAT_AP feature indicates that
> AP instructions are available on the guest. This
> feature will be enabled by the kernel only if the AP
> instructions are installed on the linux host. This feature
> must be specifically turned on for the KVM guest from
> userspace to use the VFIO AP device driver for guest
> access to AP devices.
>
> CPU model facilities:
>
> 1. AP Query Configuration Information (QCI) facility is installed.
>
> This is indicated by setting facilities bit 12 for
> the guest. The kernel will not enable this facility
> for the guest if it is not set on the host. This facility
> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
> feature is not installed.
Could it happen on real HW (not frankenstein CPU models), that
!AP, but STFL 12 or 15 are indicated?
>
> If this facility is not set for the KVM guest, then only
> APQNs with an APQI less than 16 will be used by a Linux
> guest regardless of the matrix configuration for the virtual
> machine. This is a limitation of the Linux AP bus.
>
> 2. AP Facilities Test facility (APFT) is installed.
>
> This is indicated by setting facilities bit 15 for
> the guest. The kernel will not enable this facility for
> the guest if it is not set on the host. This facility
> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
> feature is not installed.
>
> If this facility is not set for the KVM guest, then no
> AP devices will be available to the guest regardless of
> the guest's matrix configuration for the virtual
> machine. This is a limitation of the Linux AP bus.
>
> Signed-off-by: Tony Krowiak <[email protected]>
> Reviewed-by: Christian Borntraeger <[email protected]>
> Reviewed-by: Halil Pasic <[email protected]>
> Tested-by: Michael Mueller <[email protected]>
> Tested-by: Farhan Ali <[email protected]>
> Signed-off-by: Christian Borntraeger <[email protected]>
> ---
> arch/s390/kvm/kvm-s390.c | 7 +++++++
> arch/s390/tools/gen_facilities.c | 2 ++
> 2 files changed, 9 insertions(+), 0 deletions(-)
>
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 9203f0b..7d4fe9b 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -367,6 +367,13 @@ static void kvm_s390_cpu_feat_init(void)
>
> if (MACHINE_HAS_ESOP)
> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
> +
> + /*
> + * Check if AP instructions installed on host
> + */
Make this a one-line comment, please.
> + if (ap_instructions_available() == 0)
> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
> +
> /*
> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
> index 90a8c9e..a52290b 100644
> --- a/arch/s390/tools/gen_facilities.c
> +++ b/arch/s390/tools/gen_facilities.c
> @@ -106,6 +106,8 @@ struct facility_def {
>
> .name = "FACILITIES_KVM_CPUMODEL",
> .bits = (int[]){
> + 12, /* AP Query Configuration Information */
> + 15, /* AP Facilities Test */
> -1 /* END */
> }
> },
>
--
Thanks,
David / dhildenb
On 08.08.2018 16:44, Tony Krowiak wrote:
> From: Tony Krowiak <[email protected]>
>
> This patch refactors the code that initializes and sets up the
> crypto configuration for a guest. The following changes are
> implemented via this patch:
>
> 1. Prior to the introduction of AP device virtualization, it
> was not necessary to provide guest access to the CRYCB
> unless the MSA extension 3 (MSAX3) facility was installed
> on the host system. With the introduction of AP device
> virtualization, the CRYCB must be made accessible to the
> guest if the AP instructions are installed on the host
> and are to be provided to the guest.
>
> 2. Introduces a flag indicating AP instructions executed on
> the guest shall be interpreted by the firmware. It is
> initialized to indicate AP instructions are to be
> to be interpreted and is used to set the SIE bit for
> each vcpu during vcpu setup.
>
> Signed-off-by: Tony Krowiak <[email protected]>
> Reviewed-by: Halil Pasic <[email protected]>
> Acked-by: Christian Borntraeger <[email protected]>
> Tested-by: Michael Mueller <[email protected]>
> Tested-by: Farhan Ali <[email protected]>
> Signed-off-by: Christian Borntraeger <[email protected]>
> ---
> arch/s390/include/asm/kvm_host.h | 3 +
> arch/s390/include/uapi/asm/kvm.h | 1 +
> arch/s390/kvm/kvm-s390.c | 86 ++++++++++++++++++++------------------
> 3 files changed, 49 insertions(+), 41 deletions(-)
>
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index af39561..0c13f61 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -187,6 +187,7 @@ struct kvm_s390_sie_block {
> #define ECA_AIV 0x00200000
> #define ECA_VX 0x00020000
> #define ECA_PROTEXCI 0x00002000
> +#define ECA_APIE 0x00000008
> #define ECA_SII 0x00000001
> __u32 eca; /* 0x004c */
> #define ICPT_INST 0x04
> @@ -256,6 +257,7 @@ struct kvm_s390_sie_block {
> __u8 reservede4[4]; /* 0x00e4 */
> __u64 tecmc; /* 0x00e8 */
> __u8 reservedf0[12]; /* 0x00f0 */
> +#define CRYCB_FORMAT_MASK 0x00000003
> #define CRYCB_FORMAT1 0x00000001
> #define CRYCB_FORMAT2 0x00000003
> __u32 crycbd; /* 0x00fc */
> @@ -714,6 +716,7 @@ struct kvm_s390_crypto {
> __u32 crycbd;
> __u8 aes_kw;
> __u8 dea_kw;
> + __u8 apie;
This flag is essentially always 1 as far as I can see, so please drop it
for now. (can be introduced when really needed)
> };
>
> #define APCB0_MASK_SIZE 1
> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
> index 4cdaa55..a580dec 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
> #define KVM_S390_VM_CPU_FEAT_KSS 13
> +#define KVM_S390_VM_CPU_FEAT_AP 14
> struct kvm_s390_vm_cpu_feat {
> __u64 feat[16];
> };
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 19f4f44..f96a443 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -40,6 +40,7 @@
> #include <asm/sclp.h>
> #include <asm/cpacf.h>
> #include <asm/timex.h>
> +#include <asm/ap.h>
> #include "kvm-s390.h"
> #include "gaccess.h"
>
> @@ -1881,49 +1882,37 @@ long kvm_arch_vm_ioctl(struct file *filp,
> return r;
> }
>
> -static int kvm_s390_query_ap_config(u8 *config)
> -{
> - u32 fcn_code = 0x04000000UL;
> - u32 cc = 0;
> -
> - memset(config, 0, 128);
> - asm volatile(
> - "lgr 0,%1\n"
> - "lgr 2,%2\n"
> - ".long 0xb2af0000\n" /* PQAP(QCI) */
> - "0: ipm %0\n"
> - "srl %0,28\n"
> - "1:\n"
> - EX_TABLE(0b, 1b)
> - : "+r" (cc)
> - : "r" (fcn_code), "r" (config)
> - : "cc", "0", "2", "memory"
> - );
> -
> - return cc;
> -}
> -
> static int kvm_s390_apxa_installed(void)
> {
> - u8 config[128];
> - int cc;
> + struct ap_config_info info;
>
> - if (test_facility(12)) {
> - cc = kvm_s390_query_ap_config(config);
> -
> - if (cc)
> - pr_err("PQAP(QCI) failed with cc=%d", cc);
> - else
> - return config[0] & 0x40;
> + if (ap_instructions_available() == 0) {
> + if (ap_qci(&info) == 0)
> + return info.apxa;
> }
>
> return 0;
> }
>
> +/*
> + * The format of the crypto control block (CRYCB) is specified in the 3 low
> + * order bits of the CRYCB designation (CRYCBD) field as follows:
> + * Format 0: Neither the message security assist extension 3 (MSAX3) nor the
> + * AP extended addressing (APXA) facility are installed.
> + * Format 1: The APXA facility is not installed but the MSAX3 facility is.
> + * Format 2: Both the APXA and MSAX3 facilities are installed
> + */
> static void kvm_s390_set_crycb_format(struct kvm *kvm)
> {
> kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
>
> + /* Clear the CRYCB format bits - i.e., set format 0 by default */
> + kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT_MASK);
> +
> + /* Check whether MSAX3 is installed */
> + if (!test_kvm_facility(kvm, 76))
> + return;
> +
> if (kvm_s390_apxa_installed())
> kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
> else
> @@ -1941,11 +1930,13 @@ static u64 kvm_s390_get_initial_cpuid(void)
>
> static void kvm_s390_crypto_init(struct kvm *kvm)
> {
> - if (!test_kvm_facility(kvm, 76))
> - return;
> -
> kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
> kvm_s390_set_crycb_format(kvm);
> + /* Default setting indicating SIE shall interpret AP instructions */
> + kvm->arch.crypto.apie = 1;
> +
> + if (!test_kvm_facility(kvm, 76))
> + return;
>
> /* Enable AES/DEA protected key functions by default */
> kvm->arch.crypto.aes_kw = 1;
> @@ -2474,17 +2465,30 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>
> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
> {
> - if (!test_kvm_facility(vcpu->kvm, 76))
> + /*
> + * If neither the AP instructions nor the MSAX3 facility are installed
> + * on the host, then there is no need for a CRYCB in SIE because the
> + * they will not be installed on the guest either.
> + */
> + if (ap_instructions_available() && !test_facility(76))
Why not turn this into
if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
!test_kvm_facility(vcpu->kvm, 76))
These properties should no longer change at this point. And we simulate
exactly what the guest will see. No crycb needed. And even no comment
needed.
> return;
>
> - vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
> + vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> +
> + vcpu->arch.sie_block->eca &= ~ECA_APIE;
> + if (vcpu->kvm->arch.crypto.apie &&
> + test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
> + vcpu->arch.sie_block->eca |= ECA_APIE;
>
> - if (vcpu->kvm->arch.crypto.aes_kw)
> - vcpu->arch.sie_block->ecb3 |= ECB3_AES;
> - if (vcpu->kvm->arch.crypto.dea_kw)
> - vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
> + /* If MSAX3 is installed on the guest, set up protected key support */
> + if (test_kvm_facility(vcpu->kvm, 76)) {
> + vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
Why not keep the original unconditional clearing?
>
> - vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
> + if (vcpu->kvm->arch.crypto.aes_kw)
> + vcpu->arch.sie_block->ecb3 |= ECB3_AES;
> + if (vcpu->kvm->arch.crypto.dea_kw)
> + vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
> + }
> }
>
> void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
>
--
Thanks,
David / dhildenb
On 09.08.2018 10:17, David Hildenbrand wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Tony Krowiak <[email protected]>
>>
>> Introduces a new CPU model feature and two CPU model
>> facilities to support AP virtualization for KVM guests.
>>
>> CPU model feature:
>>
>> The KVM_S390_VM_CPU_FEAT_AP feature indicates that
>> AP instructions are available on the guest. This
>> feature will be enabled by the kernel only if the AP
>> instructions are installed on the linux host. This feature
>> must be specifically turned on for the KVM guest from
>> userspace to use the VFIO AP device driver for guest
>> access to AP devices.
>>
>> CPU model facilities:
>>
>> 1. AP Query Configuration Information (QCI) facility is installed.
>>
>> This is indicated by setting facilities bit 12 for
>> the guest. The kernel will not enable this facility
>> for the guest if it is not set on the host. This facility
>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>> feature is not installed.
> Could it happen on real HW (not frankenstein CPU models), that
>
> !AP, but STFL 12 or 15 are indicated?
What is your expectation here ? There are no AP instructions
available but if there would be AP instructions available then
there would be the QCI and or the T bit facility within
the PQAP AP instruction valid ?
>> If this facility is not set for the KVM guest, then only
>> APQNs with an APQI less than 16 will be used by a Linux
>> guest regardless of the matrix configuration for the virtual
>> machine. This is a limitation of the Linux AP bus.
>>
>> 2. AP Facilities Test facility (APFT) is installed.
>>
>> This is indicated by setting facilities bit 15 for
>> the guest. The kernel will not enable this facility for
>> the guest if it is not set on the host. This facility
>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>> feature is not installed.
>>
>> If this facility is not set for the KVM guest, then no
>> AP devices will be available to the guest regardless of
>> the guest's matrix configuration for the virtual
>> machine. This is a limitation of the Linux AP bus.
>>
>> Signed-off-by: Tony Krowiak <[email protected]>
>> Reviewed-by: Christian Borntraeger <[email protected]>
>> Reviewed-by: Halil Pasic <[email protected]>
>> Tested-by: Michael Mueller <[email protected]>
>> Tested-by: Farhan Ali <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
>> ---
>> arch/s390/kvm/kvm-s390.c | 7 +++++++
>> arch/s390/tools/gen_facilities.c | 2 ++
>> 2 files changed, 9 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 9203f0b..7d4fe9b 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -367,6 +367,13 @@ static void kvm_s390_cpu_feat_init(void)
>>
>> if (MACHINE_HAS_ESOP)
>> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>> +
>> + /*
>> + * Check if AP instructions installed on host
>> + */
> Make this a one-line comment, please.
>
>> + if (ap_instructions_available() == 0)
>> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
>> +
>> /*
>> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
>> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>> index 90a8c9e..a52290b 100644
>> --- a/arch/s390/tools/gen_facilities.c
>> +++ b/arch/s390/tools/gen_facilities.c
>> @@ -106,6 +106,8 @@ struct facility_def {
>>
>> .name = "FACILITIES_KVM_CPUMODEL",
>> .bits = (int[]){
>> + 12, /* AP Query Configuration Information */
>> + 15, /* AP Facilities Test */
>> -1 /* END */
>> }
>> },
>>
>
On 09/08/2018 10:10, David Hildenbrand wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Pierre Morel <[email protected]>
>>
>> When we clear the Crypto Control Block (CRYCB) used by a guest
>> level 2, the vSIE shadow CRYCB for guest level 3 must be updated
>> before the guest uses it.
>>
>> We achieve this by using the KVM_REQ_VSIE_RESTART synchronous
>> request for each vCPU belonging to the guest to force the reload
>> of the shadow CRYCB before rerunning the guest level 3.
>>
>> Signed-off-by: Pierre Morel <[email protected]>
>> ---
>> arch/s390/kvm/kvm-s390.c | 7 +++++++
>> 1 files changed, 7 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 0d03249..9203f0b 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -1921,6 +1921,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>>
>> void kvm_arch_crypto_clear_masks(struct kvm *kvm)
>> {
>> + int i;
>> + struct kvm_vcpu *vcpu;
>> +
>> mutex_lock(&kvm->lock);
>> kvm_s390_vcpu_block_all(kvm);
>>
>> @@ -1929,6 +1932,10 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm)
>> memset(&kvm->arch.crypto.crycb->apcb1, 0,
>> sizeof(kvm->arch.crypto.crycb->apcb1));
>>
>> + /* recreate the shadow crycb for each vcpu */
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
> We have kvm_s390_sync_request_broadcast(), which should boil down to a
> one-liner
right, thanks
>> +
>> kvm_s390_vcpu_unblock_all(kvm);
>> mutex_unlock(&kvm->lock);
>> }
>>
>
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
On 09/08/2018 10:10, David Hildenbrand wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Pierre Morel <[email protected]>
>>
>> When we clear the Crypto Control Block (CRYCB) used by a guest
>> level 2, the vSIE shadow CRYCB for guest level 3 must be updated
>> before the guest uses it.
>>
>> We achieve this by using the KVM_REQ_VSIE_RESTART synchronous
>> request for each vCPU belonging to the guest to force the reload
>> of the shadow CRYCB before rerunning the guest level 3.
>>
>> Signed-off-by: Pierre Morel <[email protected]>
>> ---
>> arch/s390/kvm/kvm-s390.c | 7 +++++++
>> 1 files changed, 7 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 0d03249..9203f0b 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -1921,6 +1921,9 @@ static void kvm_s390_set_crycb_format(struct kvm *kvm)
>>
>> void kvm_arch_crypto_clear_masks(struct kvm *kvm)
>> {
>> + int i;
>> + struct kvm_vcpu *vcpu;
>> +
>> mutex_lock(&kvm->lock);
>> kvm_s390_vcpu_block_all(kvm);
>>
>> @@ -1929,6 +1932,10 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm)
>> memset(&kvm->arch.crypto.crycb->apcb1, 0,
>> sizeof(kvm->arch.crypto.crycb->apcb1));
>>
>> + /* recreate the shadow crycb for each vcpu */
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
> We have kvm_s390_sync_request_broadcast(), which should boil down to a
> one-liner
do you think I should squash this patch with the next one?
>
>> +
>> kvm_s390_vcpu_unblock_all(kvm);
>> mutex_unlock(&kvm->lock);
>> }
>>
>
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
On Wed, 8 Aug 2018 10:44:14 -0400
Tony Krowiak <[email protected]> wrote:
> From: Harald Freudenberger <[email protected]>
>
> Move all the inline functions from the ap bus header
> file ap_asm.h into the in-kernel api header file
> arch/s390/include/asm/ap.h so that KVM can make use
> of all the low level AP functions.
>
> Signed-off-by: Harald Freudenberger <[email protected]>
> Signed-off-by: Christian Borntraeger <[email protected]>
You should add your own s-o-b if you are sending on patches written by
others (even if it does not matter in the end, when they are merged
through a different path anyway.)
> ---
> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
> drivers/s390/crypto/ap_bus.c | 21 +---
> drivers/s390/crypto/ap_bus.h | 1 +
> drivers/s390/crypto/ap_card.c | 1 -
> drivers/s390/crypto/ap_queue.c | 1 -
> 6 files changed, 259 insertions(+), 310 deletions(-)
> delete mode 100644 drivers/s390/crypto/ap_asm.h
>
> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
> index c1bedb4..046e044 100644
> --- a/arch/s390/include/asm/ap.h
> +++ b/arch/s390/include/asm/ap.h
> @@ -47,6 +47,50 @@ struct ap_queue_status {
> };
>
> /**
> + * ap_intructions_available() - Test if AP instructions are available.
> + *
> + * Returns 0 if the AP instructions are installed.
Stumbled over this when I was looking at the usage in patch 7: if I see
a function called '_available' return 0, I'd assume that whatever the
function tests for is *not* available.
Rather call this function ap_instructions_check_availability() (and
keep the return code convention), or switch this to return 0 if not
available and !0 if available?
> + */
> +static inline int ap_instructions_available(void)
> +{
> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
> + register unsigned long reg1 asm ("1") = -ENODEV;
> + register unsigned long reg2 asm ("2");
> +
> + asm volatile(
> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
> + "0: la %0,0\n"
> + "1:\n"
> + EX_TABLE(0b, 1b)
> + : "+d" (reg1), "=d" (reg2)
> + : "d" (reg0)
> + : "cc");
> + return reg1;
> +}
On 09.08.2018 11:06, Cornelia Huck wrote:
> On Wed, 8 Aug 2018 10:44:14 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> From: Harald Freudenberger <[email protected]>
>>
>> Move all the inline functions from the ap bus header
>> file ap_asm.h into the in-kernel api header file
>> arch/s390/include/asm/ap.h so that KVM can make use
>> of all the low level AP functions.
>>
>> Signed-off-by: Harald Freudenberger <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
> You should add your own s-o-b if you are sending on patches written by
> others (even if it does not matter in the end, when they are merged
> through a different path anyway.)
>
>> ---
>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
>> drivers/s390/crypto/ap_bus.c | 21 +---
>> drivers/s390/crypto/ap_bus.h | 1 +
>> drivers/s390/crypto/ap_card.c | 1 -
>> drivers/s390/crypto/ap_queue.c | 1 -
>> 6 files changed, 259 insertions(+), 310 deletions(-)
>> delete mode 100644 drivers/s390/crypto/ap_asm.h
>>
>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
>> index c1bedb4..046e044 100644
>> --- a/arch/s390/include/asm/ap.h
>> +++ b/arch/s390/include/asm/ap.h
>> @@ -47,6 +47,50 @@ struct ap_queue_status {
>> };
>>
>> /**
>> + * ap_intructions_available() - Test if AP instructions are available.
>> + *
>> + * Returns 0 if the AP instructions are installed.
> Stumbled over this when I was looking at the usage in patch 7: if I see
> a function called '_available' return 0, I'd assume that whatever the
> function tests for is *not* available.
>
> Rather call this function ap_instructions_check_availability() (and
> keep the return code convention), or switch this to return 0 if not
> available and !0 if available?
Good catch, Cony you are right. I'll fix this to return 1 if AP instructions
are available and 0 if not. However, this patch will come via Martin's pipe
to the Linus Torwald kernel sources.
>
>> + */
>> +static inline int ap_instructions_available(void)
>> +{
>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
>> + register unsigned long reg1 asm ("1") = -ENODEV;
>> + register unsigned long reg2 asm ("2");
>> +
>> + asm volatile(
>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
>> + "0: la %0,0\n"
>> + "1:\n"
>> + EX_TABLE(0b, 1b)
>> + : "+d" (reg1), "=d" (reg2)
>> + : "d" (reg0)
>> + : "cc");
>> + return reg1;
>> +}
> --
> To unsubscribe from this list: send the line "unsubscribe linux-s390" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Wed, 8 Aug 2018 10:44:18 -0400
Tony Krowiak <[email protected]> wrote:
> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
> new file mode 100644
> index 0000000..d7e39ad
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_drv.c
> @@ -0,0 +1,115 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * VFIO based AP device driver
> + *
> + * Copyright IBM Corp. 2018
> + *
> + * Author(s): Tony Krowiak <[email protected]>
> + * Halil Pasic <[email protected]>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/mod_devicetable.h>
> +#include <linux/slab.h>
> +#include <linux/string.h>
> +#include "vfio_ap_private.h"
> +
> +#define VFIO_AP_ROOT_NAME "vfio_ap"
> +#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
> +#define VFIO_AP_DEV_NAME "vfio_ap"
> +
> +MODULE_AUTHOR("IBM Corporation");
> +MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2018");
> +MODULE_LICENSE("GPL v2");
> +
> +static struct ap_driver vfio_ap_drv;
> +
> +static struct device_type vfio_ap_dev_type = {
> + .name = VFIO_AP_DEV_TYPE_NAME,
> +};
> +
> +struct ap_matrix_dev matrix_dev = {
> + /*
> + * We may add some fops later to implement a more program friendly
> + * management interface for vfio_ap_mdev devices.
> + */
> + .misc_dev = {
> + MISC_DYNAMIC_MINOR,
> + VFIO_AP_DEV_NAME,
> + }
Is there any reason why you embed a struct miscdevice and don't just
use it directly?
> +};
> +
> +/* Only type 10 adapters (CEX4 and later) are supported
> + * by the AP matrix device driver
> + */
> +static struct ap_device_id ap_queue_ids[] = {
> + { .dev_type = AP_DEVICE_TYPE_CEX4,
> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> + { .dev_type = AP_DEVICE_TYPE_CEX5,
> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> + { .dev_type = AP_DEVICE_TYPE_CEX6,
> + .match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
> + { /* end of sibling */ },
> +};
> +
> +MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
> +
> +static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
> +{
> + return 0;
> +}
> +
> +static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
> +{
> + /* Nothing to do yet */
> +}
> +
> +static int vfio_ap_matrix_dev_create(void)
> +{
> + int ret;
> +
> + ret = misc_register(&matrix_dev.misc_dev);
> + if (ret)
> + return ret;
> +
> + to_device(&matrix_dev)->type = &vfio_ap_dev_type;
> + to_device(&matrix_dev)->driver = &vfio_ap_drv.driver;
This looks strange. Why would you want to change the driver for a
miscdevice, and that after it already has been registered? This feels
like a miscdevice is not the right choice here.
> +
> + return ret;
> +}
> +
> +static void vfio_ap_matrix_dev_destroy(void)
> +{
> + misc_deregister(&matrix_dev.misc_dev);
> +}
> +
> +int __init vfio_ap_init(void)
> +{
> + int ret;
> +
> + ret = vfio_ap_matrix_dev_create();
> + if (ret)
> + return ret;
> +
> + memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
> + vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
> + vfio_ap_drv.remove = vfio_ap_queue_dev_remove;
> + vfio_ap_drv.ids = ap_queue_ids;
> +
> + ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
> + if (ret) {
> + vfio_ap_matrix_dev_destroy();
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +void __exit vfio_ap_exit(void)
> +{
> + ap_driver_unregister(&vfio_ap_drv);
> + vfio_ap_matrix_dev_destroy();
> +}
> +
> +module_init(vfio_ap_init);
> +module_exit(vfio_ap_exit);
> diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h
> new file mode 100644
> index 0000000..02c878c
> --- /dev/null
> +++ b/drivers/s390/crypto/vfio_ap_private.h
> @@ -0,0 +1,30 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Private data and functions for adjunct processor VFIO matrix driver.
> + *
> + * Copyright IBM Corp. 2018
> + */
> +
> +#ifndef _VFIO_AP_PRIVATE_H_
> +#define _VFIO_AP_PRIVATE_H_
> +
> +#include <linux/types.h>
> +#include <linux/miscdevice.h>
> +
> +#include "ap_bus.h"
> +
> +#define VFIO_AP_MODULE_NAME "vfio_ap"
> +#define VFIO_AP_DRV_NAME "vfio_ap"
> +
> +struct ap_matrix_dev {
> + struct miscdevice misc_dev;
> +};
> +
> +extern struct ap_matrix_dev matrix_dev;
> +
> +static inline struct device *to_device(struct ap_matrix_dev *matrix_dev)
> +{
> + return matrix_dev->misc_dev.this_device;
> +}
Even if you keep it as a miscdevice, I don't like that function,
especially the naming. Frankly, I'd simply drop it.
> +
> +#endif /* _VFIO_AP_PRIVATE_H_ */
On Wed, 8 Aug 2018 10:44:19 -0400
Tony Krowiak <[email protected]> wrote:
> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
> index d7e39ad..6a827f3 100644
> --- a/drivers/s390/crypto/vfio_ap_drv.c
> +++ b/drivers/s390/crypto/vfio_ap_drv.c
> @@ -12,6 +12,7 @@
> #include <linux/mod_devicetable.h>
> #include <linux/slab.h>
> #include <linux/string.h>
> +#include <asm/zcrypt.h>
> #include "vfio_ap_private.h"
>
> #define VFIO_AP_ROOT_NAME "vfio_ap"
> @@ -68,6 +69,18 @@ static int vfio_ap_matrix_dev_create(void)
> {
> int ret;
>
> + mutex_init(&matrix_dev.lock);
> + INIT_LIST_HEAD(&matrix_dev.mdev_list);
> +
> + /* Test if PQAP(QCI) instruction is available */
> + if (test_facility(12)) {
> + ret = ap_qci(&matrix_dev.info);
> + if (ret)
> + return ret;
> + }
> +
> + atomic_set(&matrix_dev.available_instances, MAX_ZDEV_ENTRIES_EXT);
> +
> ret = misc_register(&matrix_dev.misc_dev);
OK, you are adding more stuff other than the miscdevice after all...
still, I don't think that this is a good idea.
I think I had already asked this for a previous version: Why has this
been turned into a miscdevice? (I think my reaction to the answer was
'meh'... but I think more and more that we should not do that.)
> if (ret)
> return ret;
Here is now a reworked version of the integrate ap_asm.h into include/asm/ap.h patch
which can be used to apply it within the AP virtualisation patch queue for testing:
From c81710e7cd073c4f9a904f3539ecf17fd89c9c2d Mon Sep 17 00:00:00 2001
From: Harald Freudenberger <[email protected]>
Date: Tue, 12 Jun 2018 15:42:36 +0200
Subject: [PATCH] s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h.
Move all the inline functions from the ap bus header
file ap_asm.h into the in-kernel api header file
arch/s390/include/asm/ap.h so that KVM can make use
of all the low level AP functions.
Signed-off-by: Harald Freudenberger <[email protected]>
---
arch/s390/include/asm/ap.h | 284 +++++++++++++++++++++++++++++----
drivers/s390/crypto/ap_asm.h | 261 ------------------------------
drivers/s390/crypto/ap_bus.c | 23 +--
drivers/s390/crypto/ap_bus.h | 1 +
drivers/s390/crypto/ap_card.c | 1 -
drivers/s390/crypto/ap_queue.c | 1 -
6 files changed, 260 insertions(+), 311 deletions(-)
delete mode 100644 drivers/s390/crypto/ap_asm.h
diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
index c1bedb4c8de0..887494aa164f 100644
--- a/arch/s390/include/asm/ap.h
+++ b/arch/s390/include/asm/ap.h
@@ -46,6 +46,50 @@ struct ap_queue_status {
unsigned int _pad2 : 16;
};
+/**
+ * ap_intructions_available() - Test if AP instructions are available.
+ *
+ * Returns 1 if the AP instructions are installed, otherwise 0.
+ */
+static inline int ap_instructions_available(void)
+{
+ register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
+ register unsigned long reg1 asm ("1") = 0;
+ register unsigned long reg2 asm ("2") = 0;
+
+ asm volatile(
+ " .long 0xb2af0000\n" /* PQAP(TAPQ) */
+ "0: la %0,1\n"
+ "1:\n"
+ EX_TABLE(0b, 1b)
+ : "+d" (reg1), "+d" (reg2)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
+ * ap_tapq(): Test adjunct processor queue.
+ * @qid: The AP queue number
+ * @info: Pointer to queue descriptor
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_tapq(ap_qid_t qid, unsigned long *info)
+{
+ register unsigned long reg0 asm ("0") = qid;
+ register struct ap_queue_status reg1 asm ("1");
+ register unsigned long reg2 asm ("2");
+
+ asm volatile(".long 0xb2af0000" /* PQAP(TAPQ) */
+ : "=d" (reg1), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
+ if (info)
+ *info = reg2;
+ return reg1;
+}
+
/**
* ap_test_queue(): Test adjunct processor queue.
* @qid: The AP queue number
@@ -54,10 +98,57 @@ struct ap_queue_status {
*
* Returns AP queue status structure.
*/
-struct ap_queue_status ap_test_queue(ap_qid_t qid,
- int tbit,
- unsigned long *info);
+static inline struct ap_queue_status ap_test_queue(ap_qid_t qid,
+ int tbit,
+ unsigned long *info)
+{
+ if (tbit)
+ qid |= 1UL << 23; /* set T bit*/
+ return ap_tapq(qid, info);
+}
+/**
+ * ap_pqap_rapq(): Reset adjunct processor queue.
+ * @qid: The AP queue number
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_rapq(ap_qid_t qid)
+{
+ register unsigned long reg0 asm ("0") = qid | (1UL << 24);
+ register struct ap_queue_status reg1 asm ("1");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(RAPQ) */
+ : "=d" (reg1)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
+ * ap_pqap_zapq(): Reset and zeroize adjunct processor queue.
+ * @qid: The AP queue number
+ *
+ * Returns AP queue status structure.
+ */
+static inline struct ap_queue_status ap_zapq(ap_qid_t qid)
+{
+ register unsigned long reg0 asm ("0") = qid | (2UL << 24);
+ register struct ap_queue_status reg1 asm ("1");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(ZAPQ) */
+ : "=d" (reg1)
+ : "d" (reg0)
+ : "cc");
+ return reg1;
+}
+
+/**
+ * struct ap_config_info - convenience struct for AP crypto
+ * config info as returned by the ap_qci() function.
+ */
struct ap_config_info {
unsigned int apsc : 1; /* S bit */
unsigned int apxa : 1; /* N bit */
@@ -74,50 +165,189 @@ struct ap_config_info {
unsigned char _reserved4[16];
} __aligned(8);
-/*
- * ap_query_configuration(): Fetch cryptographic config info
+/**
+ * ap_qci(): Get AP configuration data
*
- * Returns the ap configuration info fetched via PQAP(QCI).
- * On success 0 is returned, on failure a negative errno
- * is returned, e.g. if the PQAP(QCI) instruction is not
- * available, the return value will be -EOPNOTSUPP.
+ * Returns 0 on success, or -EOPNOTSUPP.
*/
-int ap_query_configuration(struct ap_config_info *info);
+static inline int ap_qci(struct ap_config_info *config)
+{
+ register unsigned long reg0 asm ("0") = 4UL << 24;
+ register unsigned long reg1 asm ("1") = -EOPNOTSUPP;
+ register struct ap_config_info *reg2 asm ("2") = config;
+
+ asm volatile(
+ ".long 0xb2af0000\n" /* PQAP(QCI) */
+ "0: la %0,0\n"
+ "1:\n"
+ EX_TABLE(0b, 1b)
+ : "+d" (reg1)
+ : "d" (reg0), "d" (reg2)
+ : "cc", "memory");
+
+ return reg1;
+}
/*
* struct ap_qirq_ctrl - convenient struct for easy invocation
- * of the ap_queue_irq_ctrl() function. This struct is passed
- * as GR1 parameter to the PQAP(AQIC) instruction. For details
- * please see the AR documentation.
+ * of the ap_aqic() function. This struct is passed as GR1
+ * parameter to the PQAP(AQIC) instruction. For details please
+ * see the AR documentation.
*/
struct ap_qirq_ctrl {
unsigned int _res1 : 8;
- unsigned int zone : 8; /* zone info */
- unsigned int ir : 1; /* ir flag: enable (1) or disable (0) irq */
+ unsigned int zone : 8; /* zone info */
+ unsigned int ir : 1; /* ir flag: enable (1) or disable (0) irq */
unsigned int _res2 : 4;
- unsigned int gisc : 3; /* guest isc field */
+ unsigned int gisc : 3; /* guest isc field */
unsigned int _res3 : 6;
- unsigned int gf : 2; /* gisa format */
+ unsigned int gf : 2; /* gisa format */
unsigned int _res4 : 1;
- unsigned int gisa : 27; /* gisa origin */
+ unsigned int gisa : 27; /* gisa origin */
unsigned int _res5 : 1;
- unsigned int isc : 3; /* irq sub class */
+ unsigned int isc : 3; /* irq sub class */
};
/**
- * ap_queue_irq_ctrl(): Control interruption on a AP queue.
+ * ap_aqic(): Control interruption for a specific AP.
* @qid: The AP queue number
- * @qirqctrl: struct ap_qirq_ctrl, see above
+ * @qirqctrl: struct ap_qirq_ctrl (64 bit value)
* @ind: The notification indicator byte
*
* Returns AP queue status.
+ */
+static inline struct ap_queue_status ap_aqic(ap_qid_t qid,
+ struct ap_qirq_ctrl qirqctrl,
+ void *ind)
+{
+ register unsigned long reg0 asm ("0") = qid | (3UL << 24);
+ register struct ap_qirq_ctrl reg1_in asm ("1") = qirqctrl;
+ register struct ap_queue_status reg1_out asm ("1");
+ register void *reg2 asm ("2") = ind;
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(AQIC) */
+ : "=d" (reg1_out)
+ : "d" (reg0), "d" (reg1_in), "d" (reg2)
+ : "cc");
+ return reg1_out;
+}
+
+/*
+ * union ap_qact_ap_info - used together with the
+ * ap_aqic() function to provide a convenient way
+ * to handle the ap info needed by the qact function.
+ */
+union ap_qact_ap_info {
+ unsigned long val;
+ struct {
+ unsigned int : 3;
+ unsigned int mode : 3;
+ unsigned int : 26;
+ unsigned int cat : 8;
+ unsigned int : 8;
+ unsigned char ver[2];
+ };
+};
+
+/**
+ * ap_qact(): Query AP combatibility type.
+ * @qid: The AP queue number
+ * @apinfo: On input the info about the AP queue. On output the
+ * alternate AP queue info provided by the qact function
+ * in GR2 is stored in.
*
- * Control interruption on the given AP queue.
- * Just a simple wrapper function for the low level PQAP(AQIC)
- * instruction available for other kernel modules.
+ * Returns AP queue status. Check response_code field for failures.
*/
-struct ap_queue_status ap_queue_irq_ctrl(ap_qid_t qid,
- struct ap_qirq_ctrl qirqctrl,
- void *ind);
+static inline struct ap_queue_status ap_qact(ap_qid_t qid, int ifbit,
+ union ap_qact_ap_info *apinfo)
+{
+ register unsigned long reg0 asm ("0") = qid | (5UL << 24)
+ | ((ifbit & 0x01) << 22);
+ register unsigned long reg1_in asm ("1") = apinfo->val;
+ register struct ap_queue_status reg1_out asm ("1");
+ register unsigned long reg2 asm ("2");
+
+ asm volatile(
+ ".long 0xb2af0000" /* PQAP(QACT) */
+ : "+d" (reg1_in), "=d" (reg1_out), "=d" (reg2)
+ : "d" (reg0)
+ : "cc");
+ apinfo->val = reg2;
+ return reg1_out;
+}
+
+/**
+ * ap_nqap(): Send message to adjunct processor queue.
+ * @qid: The AP queue number
+ * @psmid: The program supplied message identifier
+ * @msg: The message text
+ * @length: The message length
+ *
+ * Returns AP queue status structure.
+ * Condition code 1 on NQAP can't happen because the L bit is 1.
+ * Condition code 2 on NQAP also means the send is incomplete,
+ * because a segment boundary was reached. The NQAP is repeated.
+ */
+static inline struct ap_queue_status ap_nqap(ap_qid_t qid,
+ unsigned long long psmid,
+ void *msg, size_t length)
+{
+ register unsigned long reg0 asm ("0") = qid | 0x40000000UL;
+ register struct ap_queue_status reg1 asm ("1");
+ register unsigned long reg2 asm ("2") = (unsigned long) msg;
+ register unsigned long reg3 asm ("3") = (unsigned long) length;
+ register unsigned long reg4 asm ("4") = (unsigned int) (psmid >> 32);
+ register unsigned long reg5 asm ("5") = psmid & 0xffffffff;
+
+ asm volatile (
+ "0: .long 0xb2ad0042\n" /* NQAP */
+ " brc 2,0b"
+ : "+d" (reg0), "=d" (reg1), "+d" (reg2), "+d" (reg3)
+ : "d" (reg4), "d" (reg5)
+ : "cc", "memory");
+ return reg1;
+}
+
+/**
+ * ap_dqap(): Receive message from adjunct processor queue.
+ * @qid: The AP queue number
+ * @psmid: Pointer to program supplied message identifier
+ * @msg: The message text
+ * @length: The message length
+ *
+ * Returns AP queue status structure.
+ * Condition code 1 on DQAP means the receive has taken place
+ * but only partially. The response is incomplete, hence the
+ * DQAP is repeated.
+ * Condition code 2 on DQAP also means the receive is incomplete,
+ * this time because a segment boundary was reached. Again, the
+ * DQAP is repeated.
+ * Note that gpr2 is used by the DQAP instruction to keep track of
+ * any 'residual' length, in case the instruction gets interrupted.
+ * Hence it gets zeroed before the instruction.
+ */
+static inline struct ap_queue_status ap_dqap(ap_qid_t qid,
+ unsigned long long *psmid,
+ void *msg, size_t length)
+{
+ register unsigned long reg0 asm("0") = qid | 0x80000000UL;
+ register struct ap_queue_status reg1 asm ("1");
+ register unsigned long reg2 asm("2") = 0UL;
+ register unsigned long reg4 asm("4") = (unsigned long) msg;
+ register unsigned long reg5 asm("5") = (unsigned long) length;
+ register unsigned long reg6 asm("6") = 0UL;
+ register unsigned long reg7 asm("7") = 0UL;
+
+
+ asm volatile(
+ "0: .long 0xb2ae0064\n" /* DQAP */
+ " brc 6,0b\n"
+ : "+d" (reg0), "=d" (reg1), "+d" (reg2),
+ "+d" (reg4), "+d" (reg5), "+d" (reg6), "+d" (reg7)
+ : : "cc", "memory");
+ *psmid = (((unsigned long long) reg6) << 32) + reg7;
+ return reg1;
+}
#endif /* _ASM_S390_AP_H_ */
diff --git a/drivers/s390/crypto/ap_asm.h b/drivers/s390/crypto/ap_asm.h
deleted file mode 100644
index e22ee126c9df..000000000000
--- a/drivers/s390/crypto/ap_asm.h
+++ /dev/null
@@ -1,261 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Copyright IBM Corp. 2016
- * Author(s): Martin Schwidefsky <[email protected]>
- *
- * Adjunct processor bus inline assemblies.
- */
-
-#ifndef _AP_ASM_H_
-#define _AP_ASM_H_
-
-#include <asm/isc.h>
-
-/**
- * ap_intructions_available() - Test if AP instructions are available.
- *
- * Returns 0 if the AP instructions are installed.
- */
-static inline int ap_instructions_available(void)
-{
- register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
- register unsigned long reg1 asm ("1") = -ENODEV;
- register unsigned long reg2 asm ("2");
-
- asm volatile(
- " .long 0xb2af0000\n" /* PQAP(TAPQ) */
- "0: la %0,0\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+d" (reg1), "=d" (reg2)
- : "d" (reg0)
- : "cc");
- return reg1;
-}
-
-/**
- * ap_tapq(): Test adjunct processor queue.
- * @qid: The AP queue number
- * @info: Pointer to queue descriptor
- *
- * Returns AP queue status structure.
- */
-static inline struct ap_queue_status ap_tapq(ap_qid_t qid, unsigned long *info)
-{
- register unsigned long reg0 asm ("0") = qid;
- register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm ("2");
-
- asm volatile(".long 0xb2af0000" /* PQAP(TAPQ) */
- : "=d" (reg1), "=d" (reg2)
- : "d" (reg0)
- : "cc");
- if (info)
- *info = reg2;
- return reg1;
-}
-
-/**
- * ap_pqap_rapq(): Reset adjunct processor queue.
- * @qid: The AP queue number
- *
- * Returns AP queue status structure.
- */
-static inline struct ap_queue_status ap_rapq(ap_qid_t qid)
-{
- register unsigned long reg0 asm ("0") = qid | (1UL << 24);
- register struct ap_queue_status reg1 asm ("1");
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(RAPQ) */
- : "=d" (reg1)
- : "d" (reg0)
- : "cc");
- return reg1;
-}
-
-/**
- * ap_pqap_zapq(): Reset and zeroize adjunct processor queue.
- * @qid: The AP queue number
- *
- * Returns AP queue status structure.
- */
-static inline struct ap_queue_status ap_zapq(ap_qid_t qid)
-{
- register unsigned long reg0 asm ("0") = qid | (2UL << 24);
- register struct ap_queue_status reg1 asm ("1");
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(ZAPQ) */
- : "=d" (reg1)
- : "d" (reg0)
- : "cc");
- return reg1;
-}
-
-/**
- * ap_aqic(): Control interruption for a specific AP.
- * @qid: The AP queue number
- * @qirqctrl: struct ap_qirq_ctrl (64 bit value)
- * @ind: The notification indicator byte
- *
- * Returns AP queue status.
- */
-static inline struct ap_queue_status ap_aqic(ap_qid_t qid,
- struct ap_qirq_ctrl qirqctrl,
- void *ind)
-{
- register unsigned long reg0 asm ("0") = qid | (3UL << 24);
- register struct ap_qirq_ctrl reg1_in asm ("1") = qirqctrl;
- register struct ap_queue_status reg1_out asm ("1");
- register void *reg2 asm ("2") = ind;
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(AQIC) */
- : "=d" (reg1_out)
- : "d" (reg0), "d" (reg1_in), "d" (reg2)
- : "cc");
- return reg1_out;
-}
-
-/**
- * ap_qci(): Get AP configuration data
- *
- * Returns 0 on success, or -EOPNOTSUPP.
- */
-static inline int ap_qci(void *config)
-{
- register unsigned long reg0 asm ("0") = 4UL << 24;
- register unsigned long reg1 asm ("1") = -EINVAL;
- register void *reg2 asm ("2") = (void *) config;
-
- asm volatile(
- ".long 0xb2af0000\n" /* PQAP(QCI) */
- "0: la %0,0\n"
- "1:\n"
- EX_TABLE(0b, 1b)
- : "+d" (reg1)
- : "d" (reg0), "d" (reg2)
- : "cc", "memory");
-
- return reg1;
-}
-
-/*
- * union ap_qact_ap_info - used together with the
- * ap_aqic() function to provide a convenient way
- * to handle the ap info needed by the qact function.
- */
-union ap_qact_ap_info {
- unsigned long val;
- struct {
- unsigned int : 3;
- unsigned int mode : 3;
- unsigned int : 26;
- unsigned int cat : 8;
- unsigned int : 8;
- unsigned char ver[2];
- };
-};
-
-/**
- * ap_qact(): Query AP combatibility type.
- * @qid: The AP queue number
- * @apinfo: On input the info about the AP queue. On output the
- * alternate AP queue info provided by the qact function
- * in GR2 is stored in.
- *
- * Returns AP queue status. Check response_code field for failures.
- */
-static inline struct ap_queue_status ap_qact(ap_qid_t qid, int ifbit,
- union ap_qact_ap_info *apinfo)
-{
- register unsigned long reg0 asm ("0") = qid | (5UL << 24)
- | ((ifbit & 0x01) << 22);
- register unsigned long reg1_in asm ("1") = apinfo->val;
- register struct ap_queue_status reg1_out asm ("1");
- register unsigned long reg2 asm ("2");
-
- asm volatile(
- ".long 0xb2af0000" /* PQAP(QACT) */
- : "+d" (reg1_in), "=d" (reg1_out), "=d" (reg2)
- : "d" (reg0)
- : "cc");
- apinfo->val = reg2;
- return reg1_out;
-}
-
-/**
- * ap_nqap(): Send message to adjunct processor queue.
- * @qid: The AP queue number
- * @psmid: The program supplied message identifier
- * @msg: The message text
- * @length: The message length
- *
- * Returns AP queue status structure.
- * Condition code 1 on NQAP can't happen because the L bit is 1.
- * Condition code 2 on NQAP also means the send is incomplete,
- * because a segment boundary was reached. The NQAP is repeated.
- */
-static inline struct ap_queue_status ap_nqap(ap_qid_t qid,
- unsigned long long psmid,
- void *msg, size_t length)
-{
- register unsigned long reg0 asm ("0") = qid | 0x40000000UL;
- register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm ("2") = (unsigned long) msg;
- register unsigned long reg3 asm ("3") = (unsigned long) length;
- register unsigned long reg4 asm ("4") = (unsigned int) (psmid >> 32);
- register unsigned long reg5 asm ("5") = psmid & 0xffffffff;
-
- asm volatile (
- "0: .long 0xb2ad0042\n" /* NQAP */
- " brc 2,0b"
- : "+d" (reg0), "=d" (reg1), "+d" (reg2), "+d" (reg3)
- : "d" (reg4), "d" (reg5)
- : "cc", "memory");
- return reg1;
-}
-
-/**
- * ap_dqap(): Receive message from adjunct processor queue.
- * @qid: The AP queue number
- * @psmid: Pointer to program supplied message identifier
- * @msg: The message text
- * @length: The message length
- *
- * Returns AP queue status structure.
- * Condition code 1 on DQAP means the receive has taken place
- * but only partially. The response is incomplete, hence the
- * DQAP is repeated.
- * Condition code 2 on DQAP also means the receive is incomplete,
- * this time because a segment boundary was reached. Again, the
- * DQAP is repeated.
- * Note that gpr2 is used by the DQAP instruction to keep track of
- * any 'residual' length, in case the instruction gets interrupted.
- * Hence it gets zeroed before the instruction.
- */
-static inline struct ap_queue_status ap_dqap(ap_qid_t qid,
- unsigned long long *psmid,
- void *msg, size_t length)
-{
- register unsigned long reg0 asm("0") = qid | 0x80000000UL;
- register struct ap_queue_status reg1 asm ("1");
- register unsigned long reg2 asm("2") = 0UL;
- register unsigned long reg4 asm("4") = (unsigned long) msg;
- register unsigned long reg5 asm("5") = (unsigned long) length;
- register unsigned long reg6 asm("6") = 0UL;
- register unsigned long reg7 asm("7") = 0UL;
-
-
- asm volatile(
- "0: .long 0xb2ae0064\n" /* DQAP */
- " brc 6,0b\n"
- : "+d" (reg0), "=d" (reg1), "+d" (reg2),
- "+d" (reg4), "+d" (reg5), "+d" (reg6), "+d" (reg7)
- : : "cc", "memory");
- *psmid = (((unsigned long long) reg6) << 32) + reg7;
- return reg1;
-}
-
-#endif /* _AP_ASM_H_ */
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index 35a0c2b52f82..aa1bbe6d7842 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -36,7 +36,6 @@
#include <linux/debugfs.h>
#include "ap_bus.h"
-#include "ap_asm.h"
#include "ap_debug.h"
/*
@@ -174,24 +173,6 @@ static inline int ap_qact_available(void)
return 0;
}
-/**
- * ap_test_queue(): Test adjunct processor queue.
- * @qid: The AP queue number
- * @tbit: Test facilities bit
- * @info: Pointer to queue descriptor
- *
- * Returns AP queue status structure.
- */
-struct ap_queue_status ap_test_queue(ap_qid_t qid,
- int tbit,
- unsigned long *info)
-{
- if (tbit)
- qid |= 1UL << 23; /* set T bit*/
- return ap_tapq(qid, info);
-}
-EXPORT_SYMBOL(ap_test_queue);
-
/*
* ap_query_configuration(): Fetch cryptographic config info
*
@@ -200,7 +181,7 @@ EXPORT_SYMBOL(ap_test_queue);
* is returned, e.g. if the PQAP(QCI) instruction is not
* available, the return value will be -EOPNOTSUPP.
*/
-int ap_query_configuration(struct ap_config_info *info)
+static inline int ap_query_configuration(struct ap_config_info *info)
{
if (!ap_configuration_available())
return -EOPNOTSUPP;
@@ -1220,7 +1201,7 @@ static int __init ap_module_init(void)
if (rc)
return rc;
- if (ap_instructions_available() != 0) {
+ if (!ap_instructions_available()) {
pr_warn("The hardware system does not support AP instructions\n");
return -ENODEV;
}
diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h
index 6a273c5ebca5..936541937e15 100644
--- a/drivers/s390/crypto/ap_bus.h
+++ b/drivers/s390/crypto/ap_bus.h
@@ -15,6 +15,7 @@
#include <linux/device.h>
#include <linux/types.h>
+#include <asm/isc.h>
#include <asm/ap.h>
#define AP_DEVICES 256 /* Number of AP devices. */
diff --git a/drivers/s390/crypto/ap_card.c b/drivers/s390/crypto/ap_card.c
index 2c726df210f6..c13e43292cb7 100644
--- a/drivers/s390/crypto/ap_card.c
+++ b/drivers/s390/crypto/ap_card.c
@@ -14,7 +14,6 @@
#include <asm/facility.h>
#include "ap_bus.h"
-#include "ap_asm.h"
/*
* AP card related attributes.
diff --git a/drivers/s390/crypto/ap_queue.c b/drivers/s390/crypto/ap_queue.c
index ba3a2e13b0eb..d83c1fa291ae 100644
--- a/drivers/s390/crypto/ap_queue.c
+++ b/drivers/s390/crypto/ap_queue.c
@@ -14,7 +14,6 @@
#include <asm/facility.h>
#include "ap_bus.h"
-#include "ap_asm.h"
/**
* ap_queue_irq_ctrl(): Control interruption on a AP queue.
--
2.17.1
On 08/09/2018 05:06 AM, Cornelia Huck wrote:
> On Wed, 8 Aug 2018 10:44:14 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> From: Harald Freudenberger <[email protected]>
>>
>> Move all the inline functions from the ap bus header
>> file ap_asm.h into the in-kernel api header file
>> arch/s390/include/asm/ap.h so that KVM can make use
>> of all the low level AP functions.
>>
>> Signed-off-by: Harald Freudenberger <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
> You should add your own s-o-b if you are sending on patches written by
> others (even if it does not matter in the end, when they are merged
> through a different path anyway.)
That's rather ironic given I was told in an internal review that I should
not sign off on patches I did not write.
>
>> ---
>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
>> drivers/s390/crypto/ap_bus.c | 21 +---
>> drivers/s390/crypto/ap_bus.h | 1 +
>> drivers/s390/crypto/ap_card.c | 1 -
>> drivers/s390/crypto/ap_queue.c | 1 -
>> 6 files changed, 259 insertions(+), 310 deletions(-)
>> delete mode 100644 drivers/s390/crypto/ap_asm.h
>>
>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
>> index c1bedb4..046e044 100644
>> --- a/arch/s390/include/asm/ap.h
>> +++ b/arch/s390/include/asm/ap.h
>> @@ -47,6 +47,50 @@ struct ap_queue_status {
>> };
>>
>> /**
>> + * ap_intructions_available() - Test if AP instructions are available.
>> + *
>> + * Returns 0 if the AP instructions are installed.
> Stumbled over this when I was looking at the usage in patch 7: if I see
> a function called '_available' return 0, I'd assume that whatever the
> function tests for is *not* available.
>
> Rather call this function ap_instructions_check_availability() (and
> keep the return code convention), or switch this to return 0 if not
> available and !0 if available?
>
>> + */
>> +static inline int ap_instructions_available(void)
>> +{
>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
>> + register unsigned long reg1 asm ("1") = -ENODEV;
>> + register unsigned long reg2 asm ("2");
>> +
>> + asm volatile(
>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
>> + "0: la %0,0\n"
>> + "1:\n"
>> + EX_TABLE(0b, 1b)
>> + : "+d" (reg1), "=d" (reg2)
>> + : "d" (reg0)
>> + : "cc");
>> + return reg1;
>> +}
On Thu, Aug 09, 2018 at 11:18:05AM -0400, Tony Krowiak wrote:
> On 08/09/2018 05:06 AM, Cornelia Huck wrote:
> >On Wed, 8 Aug 2018 10:44:14 -0400
> >Tony Krowiak <[email protected]> wrote:
> >
> >>From: Harald Freudenberger <[email protected]>
> >>
> >>Move all the inline functions from the ap bus header
> >>file ap_asm.h into the in-kernel api header file
> >>arch/s390/include/asm/ap.h so that KVM can make use
> >>of all the low level AP functions.
> >>
> >>Signed-off-by: Harald Freudenberger <[email protected]>
> >>Signed-off-by: Christian Borntraeger <[email protected]>
> >You should add your own s-o-b if you are sending on patches written by
> >others (even if it does not matter in the end, when they are merged
> >through a different path anyway.)
>
> That's rather ironic given I was told in an internal review that I should
> not sign off on patches I did not write.
Please point the one who said that to
Documentation/process/submitting-patches.rst / Developer's Certificate of Origin
You really should sign off patches you forward, even if you did not write
them on your own.
On 08/09/2018 05:17 AM, Harald Freudenberger wrote:
> On 09.08.2018 11:06, Cornelia Huck wrote:
>> On Wed, 8 Aug 2018 10:44:14 -0400
>> Tony Krowiak <[email protected]> wrote:
>>
>>> From: Harald Freudenberger <[email protected]>
>>>
>>> Move all the inline functions from the ap bus header
>>> file ap_asm.h into the in-kernel api header file
>>> arch/s390/include/asm/ap.h so that KVM can make use
>>> of all the low level AP functions.
>>>
>>> Signed-off-by: Harald Freudenberger <[email protected]>
>>> Signed-off-by: Christian Borntraeger <[email protected]>
>> You should add your own s-o-b if you are sending on patches written by
>> others (even if it does not matter in the end, when they are merged
>> through a different path anyway.)
>>
>>> ---
>>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
>>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
>>> drivers/s390/crypto/ap_bus.c | 21 +---
>>> drivers/s390/crypto/ap_bus.h | 1 +
>>> drivers/s390/crypto/ap_card.c | 1 -
>>> drivers/s390/crypto/ap_queue.c | 1 -
>>> 6 files changed, 259 insertions(+), 310 deletions(-)
>>> delete mode 100644 drivers/s390/crypto/ap_asm.h
>>>
>>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
>>> index c1bedb4..046e044 100644
>>> --- a/arch/s390/include/asm/ap.h
>>> +++ b/arch/s390/include/asm/ap.h
>>> @@ -47,6 +47,50 @@ struct ap_queue_status {
>>> };
>>>
>>> /**
>>> + * ap_intructions_available() - Test if AP instructions are available.
>>> + *
>>> + * Returns 0 if the AP instructions are installed.
>> Stumbled over this when I was looking at the usage in patch 7: if I see
>> a function called '_available' return 0, I'd assume that whatever the
>> function tests for is *not* available.
>>
>> Rather call this function ap_instructions_check_availability() (and
>> keep the return code convention), or switch this to return 0 if not
>> available and !0 if available?
> Good catch, Cony you are right. I'll fix this to return 1 if AP instructions
> are available and 0 if not. However, this patch will come via Martin's pipe
> to the Linus Torwald kernel sources.
Is your intent to simply indicate whether the AP instructions are
available or
not; or is the intention to indicate whether the AP instructions are
available
and if not, they why? In the former, then I agree that a boolean should be
returned; however, if the case is the latter, then what you have is fine but
maybe the function name should be changed as Connie suggests.
>>> + */
>>> +static inline int ap_instructions_available(void)
>>> +{
>>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
>>> + register unsigned long reg1 asm ("1") = -ENODEV;
>>> + register unsigned long reg2 asm ("2");
>>> +
>>> + asm volatile(
>>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
>>> + "0: la %0,0\n"
>>> + "1:\n"
>>> + EX_TABLE(0b, 1b)
>>> + : "+d" (reg1), "=d" (reg2)
>>> + : "d" (reg0)
>>> + : "cc");
>>> + return reg1;
>>> +}
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-s390" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
On 09/08/2018 13:06, Cornelia Huck wrote:
> On Wed, 8 Aug 2018 10:44:19 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
>> index d7e39ad..6a827f3 100644
>> --- a/drivers/s390/crypto/vfio_ap_drv.c
>> +++ b/drivers/s390/crypto/vfio_ap_drv.c
>> @@ -12,6 +12,7 @@
>> #include <linux/mod_devicetable.h>
>> #include <linux/slab.h>
>> #include <linux/string.h>
>> +#include <asm/zcrypt.h>
>> #include "vfio_ap_private.h"
>>
>> #define VFIO_AP_ROOT_NAME "vfio_ap"
>> @@ -68,6 +69,18 @@ static int vfio_ap_matrix_dev_create(void)
>> {
>> int ret;
>>
>> + mutex_init(&matrix_dev.lock);
>> + INIT_LIST_HEAD(&matrix_dev.mdev_list);
>> +
>> + /* Test if PQAP(QCI) instruction is available */
>> + if (test_facility(12)) {
>> + ret = ap_qci(&matrix_dev.info);
>> + if (ret)
>> + return ret;
>> + }
>> +
>> + atomic_set(&matrix_dev.available_instances, MAX_ZDEV_ENTRIES_EXT);
>> +
>> ret = misc_register(&matrix_dev.misc_dev);
> OK, you are adding more stuff other than the miscdevice after all...
> still, I don't think that this is a good idea.
>
> I think I had already asked this for a previous version: Why has this
> been turned into a miscdevice? (I think my reaction to the answer was
> 'meh'... but I think more and more that we should not do that.)
Following our off-line conversation...
I understand your concern, about the misc device and dynamically
updating the misc device structure.
For it is your main point of contention and we do not currently have
a use case for the misc device I propose we just let it go and
rebase on the device design we had in V6 and did not seem to
make any trouble to any one.
We will propose a v9 with the integration of all comment done
so far soon as possible.
Best regards,
Pierre
>
>> if (ret)
>> return ret;
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
On 08/09/2018 11:43 AM, Heiko Carstens wrote:
> On Thu, Aug 09, 2018 at 11:18:05AM -0400, Tony Krowiak wrote:
>> On 08/09/2018 05:06 AM, Cornelia Huck wrote:
>>> On Wed, 8 Aug 2018 10:44:14 -0400
>>> Tony Krowiak <[email protected]> wrote:
>>>
>>>> From: Harald Freudenberger <[email protected]>
>>>>
>>>> Move all the inline functions from the ap bus header
>>>> file ap_asm.h into the in-kernel api header file
>>>> arch/s390/include/asm/ap.h so that KVM can make use
>>>> of all the low level AP functions.
>>>>
>>>> Signed-off-by: Harald Freudenberger <[email protected]>
>>>> Signed-off-by: Christian Borntraeger <[email protected]>
>>> You should add your own s-o-b if you are sending on patches written by
>>> others (even if it does not matter in the end, when they are merged
>>> through a different path anyway.)
>> That's rather ironic given I was told in an internal review that I should
>> not sign off on patches I did not write.
> Please point the one who said that to
> Documentation/process/submitting-patches.rst / Developer's Certificate of Origin
>
> You really should sign off patches you forward, even if you did not write
> them on your own.
Duly noted and I will do so on future posts. Thanks for the reference.
On 08/09/2018 04:25 AM, David Hildenbrand wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Tony Krowiak <[email protected]>
>>
>> This patch refactors the code that initializes and sets up the
>> crypto configuration for a guest. The following changes are
>> implemented via this patch:
>>
>> 1. Prior to the introduction of AP device virtualization, it
>> was not necessary to provide guest access to the CRYCB
>> unless the MSA extension 3 (MSAX3) facility was installed
>> on the host system. With the introduction of AP device
>> virtualization, the CRYCB must be made accessible to the
>> guest if the AP instructions are installed on the host
>> and are to be provided to the guest.
>>
>> 2. Introduces a flag indicating AP instructions executed on
>> the guest shall be interpreted by the firmware. It is
>> initialized to indicate AP instructions are to be
>> to be interpreted and is used to set the SIE bit for
>> each vcpu during vcpu setup.
>>
>> Signed-off-by: Tony Krowiak <[email protected]>
>> Reviewed-by: Halil Pasic <[email protected]>
>> Acked-by: Christian Borntraeger <[email protected]>
>> Tested-by: Michael Mueller <[email protected]>
>> Tested-by: Farhan Ali <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
>> ---
>> arch/s390/include/asm/kvm_host.h | 3 +
>> arch/s390/include/uapi/asm/kvm.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 86 ++++++++++++++++++++------------------
>> 3 files changed, 49 insertions(+), 41 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index af39561..0c13f61 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -187,6 +187,7 @@ struct kvm_s390_sie_block {
>> #define ECA_AIV 0x00200000
>> #define ECA_VX 0x00020000
>> #define ECA_PROTEXCI 0x00002000
>> +#define ECA_APIE 0x00000008
>> #define ECA_SII 0x00000001
>> __u32 eca; /* 0x004c */
>> #define ICPT_INST 0x04
>> @@ -256,6 +257,7 @@ struct kvm_s390_sie_block {
>> __u8 reservede4[4]; /* 0x00e4 */
>> __u64 tecmc; /* 0x00e8 */
>> __u8 reservedf0[12]; /* 0x00f0 */
>> +#define CRYCB_FORMAT_MASK 0x00000003
>> #define CRYCB_FORMAT1 0x00000001
>> #define CRYCB_FORMAT2 0x00000003
>> __u32 crycbd; /* 0x00fc */
>> @@ -714,6 +716,7 @@ struct kvm_s390_crypto {
>> __u32 crycbd;
>> __u8 aes_kw;
>> __u8 dea_kw;
>> + __u8 apie;
>
> This flag is essentially always 1 as far as I can see, so please drop it
> for now. (can be introduced when really needed)
That would appear to be true, so it probably makes sense to drop it.
>
>> };
>>
>> #define APCB0_MASK_SIZE 1
>> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
>> index 4cdaa55..a580dec 100644
>> --- a/arch/s390/include/uapi/asm/kvm.h
>> +++ b/arch/s390/include/uapi/asm/kvm.h
>> @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {
>> #define KVM_S390_VM_CPU_FEAT_PFMFI 11
>> #define KVM_S390_VM_CPU_FEAT_SIGPIF 12
>> #define KVM_S390_VM_CPU_FEAT_KSS 13
>> +#define KVM_S390_VM_CPU_FEAT_AP 14
>> struct kvm_s390_vm_cpu_feat {
>> __u64 feat[16];
>> };
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 19f4f44..f96a443 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -40,6 +40,7 @@
>> #include <asm/sclp.h>
>> #include <asm/cpacf.h>
>> #include <asm/timex.h>
>> +#include <asm/ap.h>
>> #include "kvm-s390.h"
>> #include "gaccess.h"
>>
>> @@ -1881,49 +1882,37 @@ long kvm_arch_vm_ioctl(struct file *filp,
>> return r;
>> }
>>
>> -static int kvm_s390_query_ap_config(u8 *config)
>> -{
>> - u32 fcn_code = 0x04000000UL;
>> - u32 cc = 0;
>> -
>> - memset(config, 0, 128);
>> - asm volatile(
>> - "lgr 0,%1\n"
>> - "lgr 2,%2\n"
>> - ".long 0xb2af0000\n" /* PQAP(QCI) */
>> - "0: ipm %0\n"
>> - "srl %0,28\n"
>> - "1:\n"
>> - EX_TABLE(0b, 1b)
>> - : "+r" (cc)
>> - : "r" (fcn_code), "r" (config)
>> - : "cc", "0", "2", "memory"
>> - );
>> -
>> - return cc;
>> -}
>> -
>> static int kvm_s390_apxa_installed(void)
>> {
>> - u8 config[128];
>> - int cc;
>> + struct ap_config_info info;
>>
>> - if (test_facility(12)) {
>> - cc = kvm_s390_query_ap_config(config);
>> -
>> - if (cc)
>> - pr_err("PQAP(QCI) failed with cc=%d", cc);
>> - else
>> - return config[0] & 0x40;
>> + if (ap_instructions_available() == 0) {
>> + if (ap_qci(&info) == 0)
>> + return info.apxa;
>> }
>>
>> return 0;
>> }
>>
>> +/*
>> + * The format of the crypto control block (CRYCB) is specified in the 3 low
>> + * order bits of the CRYCB designation (CRYCBD) field as follows:
>> + * Format 0: Neither the message security assist extension 3 (MSAX3) nor the
>> + * AP extended addressing (APXA) facility are installed.
>> + * Format 1: The APXA facility is not installed but the MSAX3 facility is.
>> + * Format 2: Both the APXA and MSAX3 facilities are installed
>> + */
>> static void kvm_s390_set_crycb_format(struct kvm *kvm)
>> {
>> kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
>>
>> + /* Clear the CRYCB format bits - i.e., set format 0 by default */
>> + kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT_MASK);
>> +
>> + /* Check whether MSAX3 is installed */
>> + if (!test_kvm_facility(kvm, 76))
>> + return;
>> +
>> if (kvm_s390_apxa_installed())
>> kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
>> else
>> @@ -1941,11 +1930,13 @@ static u64 kvm_s390_get_initial_cpuid(void)
>>
>> static void kvm_s390_crypto_init(struct kvm *kvm)
>> {
>> - if (!test_kvm_facility(kvm, 76))
>> - return;
>> -
>> kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
>> kvm_s390_set_crycb_format(kvm);
>> + /* Default setting indicating SIE shall interpret AP instructions */
>> + kvm->arch.crypto.apie = 1;
>> +
>> + if (!test_kvm_facility(kvm, 76))
>> + return;
>>
>> /* Enable AES/DEA protected key functions by default */
>> kvm->arch.crypto.aes_kw = 1;
>> @@ -2474,17 +2465,30 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>>
>> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
>> {
>> - if (!test_kvm_facility(vcpu->kvm, 76))
>> + /*
>> + * If neither the AP instructions nor the MSAX3 facility are installed
>> + * on the host, then there is no need for a CRYCB in SIE because the
>> + * they will not be installed on the guest either.
>> + */
>> + if (ap_instructions_available() && !test_facility(76))
> Why not turn this into
>
> if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
> !test_kvm_facility(vcpu->kvm, 76))
>
> These properties should no longer change at this point. And we simulate
> exactly what the guest will see. No crycb needed. And even no comment
> needed.
That sounds reasonable, I will make the change.
>
>
>> return;
>>
>> - vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>> + vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> +
>> + vcpu->arch.sie_block->eca &= ~ECA_APIE;
>> + if (vcpu->kvm->arch.crypto.apie &&
>> + test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>>
>> - if (vcpu->kvm->arch.crypto.aes_kw)
>> - vcpu->arch.sie_block->ecb3 |= ECB3_AES;
>> - if (vcpu->kvm->arch.crypto.dea_kw)
>> - vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
>> + /* If MSAX3 is installed on the guest, set up protected key support */
>> + if (test_kvm_facility(vcpu->kvm, 76)) {
>> + vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
> Why not keep the original unconditional clearing?
No good reason that I can think of ... I'll restore that.
>
>>
>> - vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> + if (vcpu->kvm->arch.crypto.aes_kw)
>> + vcpu->arch.sie_block->ecb3 |= ECB3_AES;
>> + if (vcpu->kvm->arch.crypto.dea_kw)
>> + vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
>> + }
>> }
>>
>> void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
>>
>
On 08/09/2018 04:17 AM, David Hildenbrand wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Tony Krowiak <[email protected]>
>>
>> Introduces a new CPU model feature and two CPU model
>> facilities to support AP virtualization for KVM guests.
>>
>> CPU model feature:
>>
>> The KVM_S390_VM_CPU_FEAT_AP feature indicates that
>> AP instructions are available on the guest. This
>> feature will be enabled by the kernel only if the AP
>> instructions are installed on the linux host. This feature
>> must be specifically turned on for the KVM guest from
>> userspace to use the VFIO AP device driver for guest
>> access to AP devices.
>>
>> CPU model facilities:
>>
>> 1. AP Query Configuration Information (QCI) facility is installed.
>>
>> This is indicated by setting facilities bit 12 for
>> the guest. The kernel will not enable this facility
>> for the guest if it is not set on the host. This facility
>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>> feature is not installed.
> Could it happen on real HW (not frankenstein CPU models), that
>
> !AP, but STFL 12 or 15 are indicated?
I suspect that if AP instructions are not installed, then neither
STFLE.12 nor STFLE.15 would be set in the real hardware. This
comment, however, is talking about an administrator setting the
facility for a guest, not real hardware. Having said that and after
thinking about it a little more, I am going to remove that last sentence
because the CPU model configuration (e.g., as specified on the qemu
command line) will fail if the QCI or APFT facility is enabled when the
CPU model feature KVM_S390_VM_CPU_FEAT_AP is not.
>
>> If this facility is not set for the KVM guest, then only
>> APQNs with an APQI less than 16 will be used by a Linux
>> guest regardless of the matrix configuration for the virtual
>> machine. This is a limitation of the Linux AP bus.
>>
>> 2. AP Facilities Test facility (APFT) is installed.
>>
>> This is indicated by setting facilities bit 15 for
>> the guest. The kernel will not enable this facility for
>> the guest if it is not set on the host. This facility
>> must not be set by userspace if the KVM_S390_VM_CPU_FEAT_AP
>> feature is not installed.
>>
>> If this facility is not set for the KVM guest, then no
>> AP devices will be available to the guest regardless of
>> the guest's matrix configuration for the virtual
>> machine. This is a limitation of the Linux AP bus.
>>
>> Signed-off-by: Tony Krowiak <[email protected]>
>> Reviewed-by: Christian Borntraeger <[email protected]>
>> Reviewed-by: Halil Pasic <[email protected]>
>> Tested-by: Michael Mueller <[email protected]>
>> Tested-by: Farhan Ali <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
>> ---
>> arch/s390/kvm/kvm-s390.c | 7 +++++++
>> arch/s390/tools/gen_facilities.c | 2 ++
>> 2 files changed, 9 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 9203f0b..7d4fe9b 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -367,6 +367,13 @@ static void kvm_s390_cpu_feat_init(void)
>>
>> if (MACHINE_HAS_ESOP)
>> allow_cpu_feat(KVM_S390_VM_CPU_FEAT_ESOP);
>> +
>> + /*
>> + * Check if AP instructions installed on host
>> + */
> Make this a one-line comment, please.
Will do
>
>> + if (ap_instructions_available() == 0)
>> + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_AP);
>> +
>> /*
>> * We need SIE support, ESOP (PROT_READ protection for gmap_shadow),
>> * 64bit SCAO (SCA passthrough) and IDTE (for gmap_shadow unshadowing).
>> diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c
>> index 90a8c9e..a52290b 100644
>> --- a/arch/s390/tools/gen_facilities.c
>> +++ b/arch/s390/tools/gen_facilities.c
>> @@ -106,6 +106,8 @@ struct facility_def {
>>
>> .name = "FACILITIES_KVM_CPUMODEL",
>> .bits = (int[]){
>> + 12, /* AP Query Configuration Information */
>> + 15, /* AP Facilities Test */
>> -1 /* END */
>> }
>> },
>>
>
On Wed, 8 Aug 2018 10:44:27 -0400
Tony Krowiak <[email protected]> wrote:
> From: Tony Krowiak <[email protected]>
>
> Let's call PAPQ(ZAPQ) to zeroize a queue:
>
> * For each queue configured for a mediated matrix device
> when it is released.
>
> Zeroizing a queue resets the queue, clears all pending
> messages for the queue entries and disables adapter interruptions
> associated with the queue.
>
> Signed-off-by: Tony Krowiak <[email protected]>
> Reviewed-by: Halil Pasic <[email protected]>
> Tested-by: Michael Mueller <[email protected]>
> Tested-by: Farhan Ali <[email protected]>
> Signed-off-by: Christian Borntraeger <[email protected]>
> ---
> drivers/s390/crypto/vfio_ap_ops.c | 29 ++++++++++++++++++++++++++++-
> drivers/s390/crypto/vfio_ap_private.h | 25 +++++++++++++++++++++++++
> 2 files changed, 53 insertions(+), 1 deletions(-)
>
> @@ -788,7 +812,10 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
> {
> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>
> - kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> + if (matrix_mdev->kvm)
> + kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
Confused. Why is the check for matrix_mdev->kvm added here?
> +
> + vfio_ap_mdev_reset_queues(mdev, true);
> vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
> &matrix_mdev->group_notifier);
> matrix_mdev->kvm = NULL;
On Thu, 9 Aug 2018 12:06:56 -0400
Tony Krowiak <[email protected]> wrote:
> On 08/09/2018 05:17 AM, Harald Freudenberger wrote:
> > On 09.08.2018 11:06, Cornelia Huck wrote:
> >> On Wed, 8 Aug 2018 10:44:14 -0400
> >> Tony Krowiak <[email protected]> wrote:
> >>
> >>> From: Harald Freudenberger <[email protected]>
> >>>
> >>> Move all the inline functions from the ap bus header
> >>> file ap_asm.h into the in-kernel api header file
> >>> arch/s390/include/asm/ap.h so that KVM can make use
> >>> of all the low level AP functions.
> >>>
> >>> Signed-off-by: Harald Freudenberger <[email protected]>
> >>> Signed-off-by: Christian Borntraeger <[email protected]>
> >> You should add your own s-o-b if you are sending on patches written by
> >> others (even if it does not matter in the end, when they are merged
> >> through a different path anyway.)
> >>
> >>> ---
> >>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
> >>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
> >>> drivers/s390/crypto/ap_bus.c | 21 +---
> >>> drivers/s390/crypto/ap_bus.h | 1 +
> >>> drivers/s390/crypto/ap_card.c | 1 -
> >>> drivers/s390/crypto/ap_queue.c | 1 -
> >>> 6 files changed, 259 insertions(+), 310 deletions(-)
> >>> delete mode 100644 drivers/s390/crypto/ap_asm.h
> >>>
> >>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
> >>> index c1bedb4..046e044 100644
> >>> --- a/arch/s390/include/asm/ap.h
> >>> +++ b/arch/s390/include/asm/ap.h
> >>> @@ -47,6 +47,50 @@ struct ap_queue_status {
> >>> };
> >>>
> >>> /**
> >>> + * ap_intructions_available() - Test if AP instructions are available.
> >>> + *
> >>> + * Returns 0 if the AP instructions are installed.
> >> Stumbled over this when I was looking at the usage in patch 7: if I see
> >> a function called '_available' return 0, I'd assume that whatever the
> >> function tests for is *not* available.
> >>
> >> Rather call this function ap_instructions_check_availability() (and
> >> keep the return code convention), or switch this to return 0 if not
> >> available and !0 if available?
> > Good catch, Cony you are right. I'll fix this to return 1 if AP instructions
> > are available and 0 if not. However, this patch will come via Martin's pipe
> > to the Linus Torwald kernel sources.
>
> Is your intent to simply indicate whether the AP instructions are
> available or
> not; or is the intention to indicate whether the AP instructions are
> available
> and if not, they why? In the former, then I agree that a boolean should be
> returned; however, if the case is the latter, then what you have is fine but
> maybe the function name should be changed as Connie suggests.
So, can this actually fail for any reason other than "instructions not
installed"? Even if it did, the end result is that the instructions are
not usable -- I don't think the "why" would be interesting at that
point.
>
> >>> + */
> >>> +static inline int ap_instructions_available(void)
> >>> +{
> >>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
> >>> + register unsigned long reg1 asm ("1") = -ENODEV;
> >>> + register unsigned long reg2 asm ("2");
> >>> +
> >>> + asm volatile(
> >>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
> >>> + "0: la %0,0\n"
> >>> + "1:\n"
> >>> + EX_TABLE(0b, 1b)
> >>> + : "+d" (reg1), "=d" (reg2)
> >>> + : "d" (reg0)
> >>> + : "cc");
> >>> + return reg1;
> >>> +}
On Thu, 9 Aug 2018 18:27:16 +0200
Pierre Morel <[email protected]> wrote:
> On 09/08/2018 13:06, Cornelia Huck wrote:
> > On Wed, 8 Aug 2018 10:44:19 -0400
> > Tony Krowiak <[email protected]> wrote:
> >
> >> diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c
> >> index d7e39ad..6a827f3 100644
> >> --- a/drivers/s390/crypto/vfio_ap_drv.c
> >> +++ b/drivers/s390/crypto/vfio_ap_drv.c
> >> @@ -12,6 +12,7 @@
> >> #include <linux/mod_devicetable.h>
> >> #include <linux/slab.h>
> >> #include <linux/string.h>
> >> +#include <asm/zcrypt.h>
> >> #include "vfio_ap_private.h"
> >>
> >> #define VFIO_AP_ROOT_NAME "vfio_ap"
> >> @@ -68,6 +69,18 @@ static int vfio_ap_matrix_dev_create(void)
> >> {
> >> int ret;
> >>
> >> + mutex_init(&matrix_dev.lock);
> >> + INIT_LIST_HEAD(&matrix_dev.mdev_list);
> >> +
> >> + /* Test if PQAP(QCI) instruction is available */
> >> + if (test_facility(12)) {
> >> + ret = ap_qci(&matrix_dev.info);
> >> + if (ret)
> >> + return ret;
> >> + }
> >> +
> >> + atomic_set(&matrix_dev.available_instances, MAX_ZDEV_ENTRIES_EXT);
> >> +
> >> ret = misc_register(&matrix_dev.misc_dev);
> > OK, you are adding more stuff other than the miscdevice after all...
> > still, I don't think that this is a good idea.
> >
> > I think I had already asked this for a previous version: Why has this
> > been turned into a miscdevice? (I think my reaction to the answer was
> > 'meh'... but I think more and more that we should not do that.)
>
> Following our off-line conversation...
>
> I understand your concern, about the misc device and dynamically
> updating the misc device structure.
>
> For it is your main point of contention and we do not currently have
> a use case for the misc device I propose we just let it go and
> rebase on the device design we had in V6 and did not seem to
> make any trouble to any one.
>
> We will propose a v9 with the integration of all comment done
> so far soon as possible.
Sounds good to me!
On 10.08.2018 10:49, Cornelia Huck wrote:
> On Thu, 9 Aug 2018 12:06:56 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> On 08/09/2018 05:17 AM, Harald Freudenberger wrote:
>>> On 09.08.2018 11:06, Cornelia Huck wrote:
>>>> On Wed, 8 Aug 2018 10:44:14 -0400
>>>> Tony Krowiak <[email protected]> wrote:
>>>>
>>>>> From: Harald Freudenberger <[email protected]>
>>>>>
>>>>> Move all the inline functions from the ap bus header
>>>>> file ap_asm.h into the in-kernel api header file
>>>>> arch/s390/include/asm/ap.h so that KVM can make use
>>>>> of all the low level AP functions.
>>>>>
>>>>> Signed-off-by: Harald Freudenberger <[email protected]>
>>>>> Signed-off-by: Christian Borntraeger <[email protected]>
>>>> You should add your own s-o-b if you are sending on patches written by
>>>> others (even if it does not matter in the end, when they are merged
>>>> through a different path anyway.)
>>>>
>>>>> ---
>>>>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
>>>>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
>>>>> drivers/s390/crypto/ap_bus.c | 21 +---
>>>>> drivers/s390/crypto/ap_bus.h | 1 +
>>>>> drivers/s390/crypto/ap_card.c | 1 -
>>>>> drivers/s390/crypto/ap_queue.c | 1 -
>>>>> 6 files changed, 259 insertions(+), 310 deletions(-)
>>>>> delete mode 100644 drivers/s390/crypto/ap_asm.h
>>>>>
>>>>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
>>>>> index c1bedb4..046e044 100644
>>>>> --- a/arch/s390/include/asm/ap.h
>>>>> +++ b/arch/s390/include/asm/ap.h
>>>>> @@ -47,6 +47,50 @@ struct ap_queue_status {
>>>>> };
>>>>>
>>>>> /**
>>>>> + * ap_intructions_available() - Test if AP instructions are available.
>>>>> + *
>>>>> + * Returns 0 if the AP instructions are installed.
>>>> Stumbled over this when I was looking at the usage in patch 7: if I see
>>>> a function called '_available' return 0, I'd assume that whatever the
>>>> function tests for is *not* available.
>>>>
>>>> Rather call this function ap_instructions_check_availability() (and
>>>> keep the return code convention), or switch this to return 0 if not
>>>> available and !0 if available?
>>> Good catch, Cony you are right. I'll fix this to return 1 if AP instructions
>>> are available and 0 if not. However, this patch will come via Martin's pipe
>>> to the Linus Torwald kernel sources.
>> Is your intent to simply indicate whether the AP instructions are
>> available or
>> not; or is the intention to indicate whether the AP instructions are
>> available
>> and if not, they why? In the former, then I agree that a boolean should be
>> returned; however, if the case is the latter, then what you have is fine but
>> maybe the function name should be changed as Connie suggests.
> So, can this actually fail for any reason other than "instructions not
> installed"? Even if it did, the end result is that the instructions are
> not usable -- I don't think the "why" would be interesting at that
> point.
I can not think of any other reason why the PQAP(TAPQ) would fail
other than the AP instructions are not available at all. However,
the old implementation returned -ENODEV on failure and 0 on
success. The new implementation now returns 1 for success
and 0 for failure. This is just one of a couple of functions related
to ap xxx available. I'll rework them all to return a boolean value
soon.
>
>>>>> + */
>>>>> +static inline int ap_instructions_available(void)
>>>>> +{
>>>>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
>>>>> + register unsigned long reg1 asm ("1") = -ENODEV;
>>>>> + register unsigned long reg2 asm ("2");
>>>>> +
>>>>> + asm volatile(
>>>>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
>>>>> + "0: la %0,0\n"
>>>>> + "1:\n"
>>>>> + EX_TABLE(0b, 1b)
>>>>> + : "+d" (reg1), "=d" (reg2)
>>>>> + : "d" (reg0)
>>>>> + : "cc");
>>>>> + return reg1;
>>>>> +}
On 10/08/2018 11:14, Cornelia Huck wrote:
> On Wed, 8 Aug 2018 10:44:27 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> From: Tony Krowiak <[email protected]>
>>
>> Let's call PAPQ(ZAPQ) to zeroize a queue:
>>
>> * For each queue configured for a mediated matrix device
>> when it is released.
>>
>> Zeroizing a queue resets the queue, clears all pending
>> messages for the queue entries and disables adapter interruptions
>> associated with the queue.
>>
>> Signed-off-by: Tony Krowiak <[email protected]>
>> Reviewed-by: Halil Pasic <[email protected]>
>> Tested-by: Michael Mueller <[email protected]>
>> Tested-by: Farhan Ali <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
>> ---
>> drivers/s390/crypto/vfio_ap_ops.c | 29 ++++++++++++++++++++++++++++-
>> drivers/s390/crypto/vfio_ap_private.h | 25 +++++++++++++++++++++++++
>> 2 files changed, 53 insertions(+), 1 deletions(-)
>>
>> @@ -788,7 +812,10 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
>> {
>> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>
>> - kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>> + if (matrix_mdev->kvm)
>> + kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> Confused. Why is the check for matrix_mdev->kvm added here?
When using the KVM notifier we can get two notifications:
-> KVM is here / is comming
-> KVM is not here / disappearing
In the first case we initialize matrix_mdev->kvm with a pointer to KVM
In the second case we nullify the pointer.
During the open of the mediated device, the guest should have been started
or we refuse to start.
During the close of the mediated device, the guest should be there, but
we have no certitude that the guest did not disappear before the VFIO
file being closed.
Since we do not allow multiple guests using the same mediated device
this case should not happen with QEMU. But I am not sure that
a rogue user program could not stop KVM before closing the VFIO
mediated device.
Maybe Alex can confirm this point, if not we can remove the test.
Thanks
Pierre
>
>> +
>> + vfio_ap_mdev_reset_queues(mdev, true);
>> vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
>> &matrix_mdev->group_notifier);
>> matrix_mdev->kvm = NULL;
--
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
On Fri, 10 Aug 2018 12:49:08 +0200
Pierre Morel <[email protected]> wrote:
> On 10/08/2018 11:14, Cornelia Huck wrote:
> > On Wed, 8 Aug 2018 10:44:27 -0400
> > Tony Krowiak <[email protected]> wrote:
> >
> >> From: Tony Krowiak <[email protected]>
> >>
> >> Let's call PAPQ(ZAPQ) to zeroize a queue:
> >>
> >> * For each queue configured for a mediated matrix device
> >> when it is released.
> >>
> >> Zeroizing a queue resets the queue, clears all pending
> >> messages for the queue entries and disables adapter interruptions
> >> associated with the queue.
> >>
> >> Signed-off-by: Tony Krowiak <[email protected]>
> >> Reviewed-by: Halil Pasic <[email protected]>
> >> Tested-by: Michael Mueller <[email protected]>
> >> Tested-by: Farhan Ali <[email protected]>
> >> Signed-off-by: Christian Borntraeger <[email protected]>
> >> ---
> >> drivers/s390/crypto/vfio_ap_ops.c | 29 ++++++++++++++++++++++++++++-
> >> drivers/s390/crypto/vfio_ap_private.h | 25 +++++++++++++++++++++++++
> >> 2 files changed, 53 insertions(+), 1 deletions(-)
> >>
> >> @@ -788,7 +812,10 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
> >> {
> >> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> >>
> >> - kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> >> + if (matrix_mdev->kvm)
> >> + kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> > Confused. Why is the check for matrix_mdev->kvm added here?
>
> When using the KVM notifier we can get two notifications:
> -> KVM is here / is comming
> -> KVM is not here / disappearing
>
> In the first case we initialize matrix_mdev->kvm with a pointer to KVM
> In the second case we nullify the pointer.
>
> During the open of the mediated device, the guest should have been started
> or we refuse to start.
>
> During the close of the mediated device, the guest should be there, but
> we have no certitude that the guest did not disappear before the VFIO
> file being closed.
> Since we do not allow multiple guests using the same mediated device
> this case should not happen with QEMU. But I am not sure that
> a rogue user program could not stop KVM before closing the VFIO
> mediated device.
I'm not sure why the check is introduced in this patch, though. But
maybe I just need weekend :)
>
> Maybe Alex can confirm this point, if not we can remove the test.
On 08/10/2018 05:37 AM, Harald Freudenberger wrote:
> On 10.08.2018 10:49, Cornelia Huck wrote:
>> On Thu, 9 Aug 2018 12:06:56 -0400
>> Tony Krowiak <[email protected]> wrote:
>>
>>> On 08/09/2018 05:17 AM, Harald Freudenberger wrote:
>>>> On 09.08.2018 11:06, Cornelia Huck wrote:
>>>>> On Wed, 8 Aug 2018 10:44:14 -0400
>>>>> Tony Krowiak <[email protected]> wrote:
>>>>>
>>>>>> From: Harald Freudenberger <[email protected]>
>>>>>>
>>>>>> Move all the inline functions from the ap bus header
>>>>>> file ap_asm.h into the in-kernel api header file
>>>>>> arch/s390/include/asm/ap.h so that KVM can make use
>>>>>> of all the low level AP functions.
>>>>>>
>>>>>> Signed-off-by: Harald Freudenberger <[email protected]>
>>>>>> Signed-off-by: Christian Borntraeger <[email protected]>
>>>>> You should add your own s-o-b if you are sending on patches written by
>>>>> others (even if it does not matter in the end, when they are merged
>>>>> through a different path anyway.)
>>>>>
>>>>>> ---
>>>>>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
>>>>>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
>>>>>> drivers/s390/crypto/ap_bus.c | 21 +---
>>>>>> drivers/s390/crypto/ap_bus.h | 1 +
>>>>>> drivers/s390/crypto/ap_card.c | 1 -
>>>>>> drivers/s390/crypto/ap_queue.c | 1 -
>>>>>> 6 files changed, 259 insertions(+), 310 deletions(-)
>>>>>> delete mode 100644 drivers/s390/crypto/ap_asm.h
>>>>>>
>>>>>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
>>>>>> index c1bedb4..046e044 100644
>>>>>> --- a/arch/s390/include/asm/ap.h
>>>>>> +++ b/arch/s390/include/asm/ap.h
>>>>>> @@ -47,6 +47,50 @@ struct ap_queue_status {
>>>>>> };
>>>>>>
>>>>>> /**
>>>>>> + * ap_intructions_available() - Test if AP instructions are available.
>>>>>> + *
>>>>>> + * Returns 0 if the AP instructions are installed.
>>>>> Stumbled over this when I was looking at the usage in patch 7: if I see
>>>>> a function called '_available' return 0, I'd assume that whatever the
>>>>> function tests for is *not* available.
>>>>>
>>>>> Rather call this function ap_instructions_check_availability() (and
>>>>> keep the return code convention), or switch this to return 0 if not
>>>>> available and !0 if available?
>>>> Good catch, Cony you are right. I'll fix this to return 1 if AP instructions
>>>> are available and 0 if not. However, this patch will come via Martin's pipe
>>>> to the Linus Torwald kernel sources.
>>> Is your intent to simply indicate whether the AP instructions are
>>> available or
>>> not; or is the intention to indicate whether the AP instructions are
>>> available
>>> and if not, they why? In the former, then I agree that a boolean should be
>>> returned; however, if the case is the latter, then what you have is fine but
>>> maybe the function name should be changed as Connie suggests.
>> So, can this actually fail for any reason other than "instructions not
>> installed"? Even if it did, the end result is that the instructions are
>> not usable -- I don't think the "why" would be interesting at that
>> point.
> I can not think of any other reason why the PQAP(TAPQ) would fail
> other than the AP instructions are not available at all. However,
> the old implementation returned -ENODEV on failure and 0 on
> success. The new implementation now returns 1 for success
> and 0 for failure. This is just one of a couple of functions related
> to ap xxx available. I'll rework them all to return a boolean value
> soon.
How would you recommend I proceed given I have functions that call this
interface that check the rc and I've had to include your patches in
this series because of that dependence?
>>>>>> + */
>>>>>> +static inline int ap_instructions_available(void)
>>>>>> +{
>>>>>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
>>>>>> + register unsigned long reg1 asm ("1") = -ENODEV;
>>>>>> + register unsigned long reg2 asm ("2");
>>>>>> +
>>>>>> + asm volatile(
>>>>>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
>>>>>> + "0: la %0,0\n"
>>>>>> + "1:\n"
>>>>>> + EX_TABLE(0b, 1b)
>>>>>> + : "+d" (reg1), "=d" (reg2)
>>>>>> + : "d" (reg0)
>>>>>> + : "cc");
>>>>>> + return reg1;
>>>>>> +}
On 08/10/2018 04:49 AM, Cornelia Huck wrote:
> On Thu, 9 Aug 2018 12:06:56 -0400
> Tony Krowiak <[email protected]> wrote:
>
>> On 08/09/2018 05:17 AM, Harald Freudenberger wrote:
>>> On 09.08.2018 11:06, Cornelia Huck wrote:
>>>> On Wed, 8 Aug 2018 10:44:14 -0400
>>>> Tony Krowiak <[email protected]> wrote:
>>>>
>>>>> From: Harald Freudenberger <[email protected]>
>>>>>
>>>>> Move all the inline functions from the ap bus header
>>>>> file ap_asm.h into the in-kernel api header file
>>>>> arch/s390/include/asm/ap.h so that KVM can make use
>>>>> of all the low level AP functions.
>>>>>
>>>>> Signed-off-by: Harald Freudenberger <[email protected]>
>>>>> Signed-off-by: Christian Borntraeger <[email protected]>
>>>> You should add your own s-o-b if you are sending on patches written by
>>>> others (even if it does not matter in the end, when they are merged
>>>> through a different path anyway.)
>>>>
>>>>> ---
>>>>> arch/s390/include/asm/ap.h | 284 ++++++++++++++++++++++++++++++++++++----
>>>>> drivers/s390/crypto/ap_asm.h | 261 ------------------------------------
>>>>> drivers/s390/crypto/ap_bus.c | 21 +---
>>>>> drivers/s390/crypto/ap_bus.h | 1 +
>>>>> drivers/s390/crypto/ap_card.c | 1 -
>>>>> drivers/s390/crypto/ap_queue.c | 1 -
>>>>> 6 files changed, 259 insertions(+), 310 deletions(-)
>>>>> delete mode 100644 drivers/s390/crypto/ap_asm.h
>>>>>
>>>>> diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
>>>>> index c1bedb4..046e044 100644
>>>>> --- a/arch/s390/include/asm/ap.h
>>>>> +++ b/arch/s390/include/asm/ap.h
>>>>> @@ -47,6 +47,50 @@ struct ap_queue_status {
>>>>> };
>>>>>
>>>>> /**
>>>>> + * ap_intructions_available() - Test if AP instructions are available.
>>>>> + *
>>>>> + * Returns 0 if the AP instructions are installed.
>>>> Stumbled over this when I was looking at the usage in patch 7: if I see
>>>> a function called '_available' return 0, I'd assume that whatever the
>>>> function tests for is *not* available.
>>>>
>>>> Rather call this function ap_instructions_check_availability() (and
>>>> keep the return code convention), or switch this to return 0 if not
>>>> available and !0 if available?
>>> Good catch, Cony you are right. I'll fix this to return 1 if AP instructions
>>> are available and 0 if not. However, this patch will come via Martin's pipe
>>> to the Linus Torwald kernel sources.
>> Is your intent to simply indicate whether the AP instructions are
>> available or
>> not; or is the intention to indicate whether the AP instructions are
>> available
>> and if not, they why? In the former, then I agree that a boolean should be
>> returned; however, if the case is the latter, then what you have is fine but
>> maybe the function name should be changed as Connie suggests.
> So, can this actually fail for any reason other than "instructions not
> installed"? Even if it did, the end result is that the instructions are
> not usable -- I don't think the "why" would be interesting at that
> point.
The only case I can think of is if something is hosed and it causes an
exception. In that case, should we proceed?
>
>>>>> + */
>>>>> +static inline int ap_instructions_available(void)
>>>>> +{
>>>>> + register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
>>>>> + register unsigned long reg1 asm ("1") = -ENODEV;
>>>>> + register unsigned long reg2 asm ("2");
>>>>> +
>>>>> + asm volatile(
>>>>> + " .long 0xb2af0000\n" /* PQAP(TAPQ) */
>>>>> + "0: la %0,0\n"
>>>>> + "1:\n"
>>>>> + EX_TABLE(0b, 1b)
>>>>> + : "+d" (reg1), "=d" (reg2)
>>>>> + : "d" (reg0)
>>>>> + : "cc");
>>>>> + return reg1;
>>>>> +}
On 08/09/2018 01:58 AM, Janosch Frank wrote:
> On 08.08.2018 16:44, Tony Krowiak wrote:
>> From: Tony Krowiak <[email protected]>
>>
>> This patch refactors the code that initializes and sets up the
>> crypto configuration for a guest. The following changes are
>> implemented via this patch:
>>
>> 1. Prior to the introduction of AP device virtualization, it
>> was not necessary to provide guest access to the CRYCB
>> unless the MSA extension 3 (MSAX3) facility was installed
>> on the host system. With the introduction of AP device
>> virtualization, the CRYCB must be made accessible to the
>> guest if the AP instructions are installed on the host
>> and are to be provided to the guest.
>>
>> 2. Introduces a flag indicating AP instructions executed on
>> the guest shall be interpreted by the firmware. It is
>> initialized to indicate AP instructions are to be
>> to be interpreted and is used to set the SIE bit for
>> each vcpu during vcpu setup.
>>
>> Signed-off-by: Tony Krowiak <[email protected]>
>> Reviewed-by: Halil Pasic <[email protected]>
>> Acked-by: Christian Borntraeger <[email protected]>
>> Tested-by: Michael Mueller <[email protected]>
>> Tested-by: Farhan Ali <[email protected]>
>> Signed-off-by: Christian Borntraeger <[email protected]>
> Acked-by: Janosch Frank <[email protected]>
>
>> ---
>> arch/s390/include/asm/kvm_host.h | 3 +
>> arch/s390/include/uapi/asm/kvm.h | 1 +
>> arch/s390/kvm/kvm-s390.c | 86 ++++++++++++++++++++------------------
>> 3 files changed, 49 insertions(+), 41 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index af39561..0c13f61 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -187,6 +187,7 @@ struct kvm_s390_sie_block {
>> #define ECA_AIV 0x00200000
>> #define ECA_VX 0x00020000
>> #define ECA_PROTEXCI 0x00002000
>> +#define ECA_APIE 0x00000008
>> #define ECA_SII 0x00000001
>> __u32 eca; /* 0x004c */
>> #define ICPT_INST 0x04
>> @@ -256,6 +257,7 @@ struct kvm_s390_sie_block {
>> __u8 reservede4[4]; /* 0x00e4 */
>> __u64 tecmc; /* 0x00e8 */
>> __u8 reservedf0[12]; /* 0x00f0 */
>> +#define CRYCB_FORMAT_MASK 0x00000003
>> #define CRYCB_FORMAT1 0x00000001
>> #define CRYCB_FORMAT2 0x00000003
>> __u32 crycbd; /* 0x00fc */
>> @@ -714,6 +716,7 @@ struct kvm_s390_crypto {
>> __u32 crycbd;
>> __u8 aes_kw;
>> __u8 dea_kw;
>> + __u8 apie;
> In the last review I wanted a comment here to know what they do.
I'm not sure what the 'they' are that you reference here. I couldn't
find your review comment but I assume you are looking for a comment
explaining what the 'apie' field is used for. I am removing
the 'apie' field based on a review comment by David, so there is
no longer a need for a comment here, assuming that is what you
are referring to.
>
>> static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
>> {
>> - if (!test_kvm_facility(vcpu->kvm, 76))
>> + /*
>> + * If neither the AP instructions nor the MSAX3 facility are installed
>> + * on the host, then there is no need for a CRYCB in SIE because the
>> + * they will not be installed on the guest either.
> the they
I'll fix this grammatical error.
>
>> + */
>> + if (ap_instructions_available() && !test_facility(76))
>> return;
> I know you're not responsible for that one :) but 0 being the wanted
> value here is a bit counter-intuitive.
Based on another review comment by David, I've changed this to:
if (!test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP) &&
!test_kvm_facility(vcpu->kvm, 76))
>
>>
>> - vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>> + vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> +
>> + vcpu->arch.sie_block->eca &= ~ECA_APIE;
> The scb is zero allocated, are the ECA and the ECB3s set somewhere
> in-between, or is that your way of making sure the controls are
> definitely gone for good?
It is a bit of defensive programming. There is a KVM_SET_DEVICE_ATTR
ioctl to set crypto attributes (KVM_S390_VM_CRYPTO) that ultimately
calls the kvm_s390_vm_set_crypto() function. You'll notice that at the
end of that function, there is a call to kvm_s390_vcpu_crypto_reset_all()
that calls the kvm_s390_vcpu_crypto_setup() for each vcpu, so it would
seem that there is the possiblility that there could be a need to
set ECB3 to a different value. At any rate, I see no good reason to
remove this.
>
>> + if (vcpu->kvm->arch.crypto.apie &&
>> + test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_AP))
>> + vcpu->arch.sie_block->eca |= ECA_APIE;
>>
>> - if (vcpu->kvm->arch.crypto.aes_kw)
>> - vcpu->arch.sie_block->ecb3 |= ECB3_AES;
>> - if (vcpu->kvm->arch.crypto.dea_kw)
>> - vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
>> + /* If MSAX3 is installed on the guest, set up protected key support */
>> + if (test_kvm_facility(vcpu->kvm, 76)) {
>> + vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
>>
>> - vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
>> + if (vcpu->kvm->arch.crypto.aes_kw)
>> + vcpu->arch.sie_block->ecb3 |= ECB3_AES;
>> + if (vcpu->kvm->arch.crypto.dea_kw)
>> + vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
>> + }
>> }
>>
>> void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
>>
>
On 08/10/2018 07:16 AM, Cornelia Huck wrote:
> On Fri, 10 Aug 2018 12:49:08 +0200
> Pierre Morel <[email protected]> wrote:
>
>> On 10/08/2018 11:14, Cornelia Huck wrote:
>>> On Wed, 8 Aug 2018 10:44:27 -0400
>>> Tony Krowiak <[email protected]> wrote:
>>>
>>>> From: Tony Krowiak <[email protected]>
>>>>
>>>> Let's call PAPQ(ZAPQ) to zeroize a queue:
>>>>
>>>> * For each queue configured for a mediated matrix device
>>>> when it is released.
>>>>
>>>> Zeroizing a queue resets the queue, clears all pending
>>>> messages for the queue entries and disables adapter interruptions
>>>> associated with the queue.
>>>>
>>>> Signed-off-by: Tony Krowiak <[email protected]>
>>>> Reviewed-by: Halil Pasic <[email protected]>
>>>> Tested-by: Michael Mueller <[email protected]>
>>>> Tested-by: Farhan Ali <[email protected]>
>>>> Signed-off-by: Christian Borntraeger <[email protected]>
>>>> ---
>>>> drivers/s390/crypto/vfio_ap_ops.c | 29 ++++++++++++++++++++++++++++-
>>>> drivers/s390/crypto/vfio_ap_private.h | 25 +++++++++++++++++++++++++
>>>> 2 files changed, 53 insertions(+), 1 deletions(-)
>>>>
>>>> @@ -788,7 +812,10 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
>>>> {
>>>> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>>>
>>>> - kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>>>> + if (matrix_mdev->kvm)
>>>> + kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>>> Confused. Why is the check for matrix_mdev->kvm added here?
>> When using the KVM notifier we can get two notifications:
>> -> KVM is here / is comming
>> -> KVM is not here / disappearing
>>
>> In the first case we initialize matrix_mdev->kvm with a pointer to KVM
>> In the second case we nullify the pointer.
>>
>> During the open of the mediated device, the guest should have been started
>> or we refuse to start.
>>
>> During the close of the mediated device, the guest should be there, but
>> we have no certitude that the guest did not disappear before the VFIO
>> file being closed.
>> Since we do not allow multiple guests using the same mediated device
>> this case should not happen with QEMU. But I am not sure that
>> a rogue user program could not stop KVM before closing the VFIO
>> mediated device.
> I'm not sure why the check is introduced in this patch, though. But
> maybe I just need weekend :)
Good catch, it belongs in patch 15 where the function is introduced.
Is that the only reason for your objection?
>
>> Maybe Alex can confirm this point, if not we can remove the test.
On Fri, 10 Aug 2018 12:24:47 -0400
Tony Krowiak <[email protected]> wrote:
> On 08/10/2018 07:16 AM, Cornelia Huck wrote:
> > On Fri, 10 Aug 2018 12:49:08 +0200
> > Pierre Morel <[email protected]> wrote:
> >
> >> On 10/08/2018 11:14, Cornelia Huck wrote:
> >>> On Wed, 8 Aug 2018 10:44:27 -0400
> >>> Tony Krowiak <[email protected]> wrote:
> >>>
> >>>> From: Tony Krowiak <[email protected]>
> >>>>
> >>>> Let's call PAPQ(ZAPQ) to zeroize a queue:
> >>>>
> >>>> * For each queue configured for a mediated matrix device
> >>>> when it is released.
> >>>>
> >>>> Zeroizing a queue resets the queue, clears all pending
> >>>> messages for the queue entries and disables adapter interruptions
> >>>> associated with the queue.
> >>>>
> >>>> Signed-off-by: Tony Krowiak <[email protected]>
> >>>> Reviewed-by: Halil Pasic <[email protected]>
> >>>> Tested-by: Michael Mueller <[email protected]>
> >>>> Tested-by: Farhan Ali <[email protected]>
> >>>> Signed-off-by: Christian Borntraeger <[email protected]>
> >>>> ---
> >>>> drivers/s390/crypto/vfio_ap_ops.c | 29 ++++++++++++++++++++++++++++-
> >>>> drivers/s390/crypto/vfio_ap_private.h | 25 +++++++++++++++++++++++++
> >>>> 2 files changed, 53 insertions(+), 1 deletions(-)
> >>>>
> >>>> @@ -788,7 +812,10 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev)
> >>>> {
> >>>> struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
> >>>>
> >>>> - kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> >>>> + if (matrix_mdev->kvm)
> >>>> + kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
> >>> Confused. Why is the check for matrix_mdev->kvm added here?
> >> When using the KVM notifier we can get two notifications:
> >> -> KVM is here / is comming
> >> -> KVM is not here / disappearing
> >>
> >> In the first case we initialize matrix_mdev->kvm with a pointer to KVM
> >> In the second case we nullify the pointer.
> >>
> >> During the open of the mediated device, the guest should have been started
> >> or we refuse to start.
> >>
> >> During the close of the mediated device, the guest should be there, but
> >> we have no certitude that the guest did not disappear before the VFIO
> >> file being closed.
> >> Since we do not allow multiple guests using the same mediated device
> >> this case should not happen with QEMU. But I am not sure that
> >> a rogue user program could not stop KVM before closing the VFIO
> >> mediated device.
> > I'm not sure why the check is introduced in this patch, though. But
> > maybe I just need weekend :)
>
> Good catch, it belongs in patch 15 where the function is introduced.
> Is that the only reason for your objection?
Yes, this is what confused me. Moving this to patch 15 sounds like a
good idea :)
>
> >
> >> Maybe Alex can confirm this point, if not we can remove the test.
>
>
Here is now the add-on patch which changes the returncode for the
ap_instructions_available() function.
I talked with Heiko Carstens and this patch will go into the next pull
request for the 4.19 kernel.
-----------------------------------------------------------------------------------------
From 9e050f843f3281da1f65292422e30f2dd1fd6d98 Mon Sep 17 00:00:00 2001
From: Harald Freudenberger <[email protected]>
Date: Thu, 9 Aug 2018 11:59:34 +0200
Subject: [PATCH] s390/zcrypt: fix ap_instructions_available() returncodes
During review of KVM patches it was complained that the
ap_instructions_available() function returns 0 if AP
instructions are available and -ENODEV if not. The function
acts like a boolean function to check for AP instructions
available and thus should return 0 on failure and != 0 on
success. Changed to the suggested behaviour and adapted
the one and only caller of this function which is the ap
bus core code.
Signed-off-by: Harald Freudenberger <[email protected]>
---
arch/s390/include/asm/ap.h | 10 +++++-----
drivers/s390/crypto/ap_bus.c | 2 +-
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
index 046e044a48d0..887494aa164f 100644
--- a/arch/s390/include/asm/ap.h
+++ b/arch/s390/include/asm/ap.h
@@ -49,20 +49,20 @@ struct ap_queue_status {
/**
* ap_intructions_available() - Test if AP instructions are available.
*
- * Returns 0 if the AP instructions are installed.
+ * Returns 1 if the AP instructions are installed, otherwise 0.
*/
static inline int ap_instructions_available(void)
{
register unsigned long reg0 asm ("0") = AP_MKQID(0, 0);
- register unsigned long reg1 asm ("1") = -ENODEV;
- register unsigned long reg2 asm ("2");
+ register unsigned long reg1 asm ("1") = 0;
+ register unsigned long reg2 asm ("2") = 0;
asm volatile(
" .long 0xb2af0000\n" /* PQAP(TAPQ) */
- "0: la %0,0\n"
+ "0: la %0,1\n"
"1:\n"
EX_TABLE(0b, 1b)
- : "+d" (reg1), "=d" (reg2)
+ : "+d" (reg1), "+d" (reg2)
: "d" (reg0)
: "cc");
return reg1;
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index b023e35786ec..b1b73945d746 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -1475,7 +1475,7 @@ static int __init ap_module_init(void)
if (rc)
return rc;
- if (ap_instructions_available() != 0) {
+ if (!ap_instructions_available()) {
pr_warn("The hardware system does not support AP instructions\n");
return -ENODEV;
}
--
2.17.1
On Mon, 13 Aug 2018 11:24:48 +0200
Harald Freudenberger <[email protected]> wrote:
> Here is now the add-on patch which changes the returncode for the
> ap_instructions_available() function.
>
> I talked with Heiko Carstens and this patch will go into the next pull
> request for the 4.19 kernel.
I was just about to ask :)
>
> -----------------------------------------------------------------------------------------
> From 9e050f843f3281da1f65292422e30f2dd1fd6d98 Mon Sep 17 00:00:00 2001
> From: Harald Freudenberger <[email protected]>
> Date: Thu, 9 Aug 2018 11:59:34 +0200
> Subject: [PATCH] s390/zcrypt: fix ap_instructions_available() returncodes
>
> During review of KVM patches it was complained that the
> ap_instructions_available() function returns 0 if AP
> instructions are available and -ENODEV if not. The function
> acts like a boolean function to check for AP instructions
> available and thus should return 0 on failure and != 0 on
> success. Changed to the suggested behaviour and adapted
> the one and only caller of this function which is the ap
> bus core code.
>
> Signed-off-by: Harald Freudenberger <[email protected]>
> ---
> arch/s390/include/asm/ap.h | 10 +++++-----
> drivers/s390/crypto/ap_bus.c | 2 +-
> 2 files changed, 6 insertions(+), 6 deletions(-)
Looks sane to me.
Acked-by: Cornelia Huck <[email protected]>