DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.
The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.
A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).
All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.
The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.
Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.
More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst
The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
driver. The changes were needed in order to re-use code and to export
some more functions that are needed by the VFIO driver.
Currenlty the mc-bus patches are under review:
https://www.spinics.net/lists/kernel/msg3447567.html
Bharat Bhushan (1):
vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices
Diana Craciun (8):
vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
vfio/fsl-mc: Added lock support in preparation for interrupt handling
vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
vfio/fsl-mc: trigger an interrupt via eventfd
vfio/fsl-mc: Add read/write support for fsl-mc devices
MAINTAINERS | 6 +
drivers/vfio/Kconfig | 1 +
drivers/vfio/Makefile | 1 +
drivers/vfio/fsl-mc/Kconfig | 9 +
drivers/vfio/fsl-mc/Makefile | 4 +
drivers/vfio/fsl-mc/vfio_fsl_mc.c | 660 ++++++++++++++++++++++
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 221 ++++++++
drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 56 ++
include/uapi/linux/vfio.h | 1 +
9 files changed, 959 insertions(+)
create mode 100644 drivers/vfio/fsl-mc/Kconfig
create mode 100644 drivers/vfio/fsl-mc/Makefile
create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
--
2.17.1
The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.
Signed-off-by: Bharat Bhushan <[email protected]>
Signed-off-by: Diana Craciun <[email protected]>
---
drivers/vfio/fsl-mc/vfio_fsl_mc.c | 66 +++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 320fb09b5691..5cc533808bc1 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -74,6 +74,34 @@ static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma)
return -EINVAL;
}
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+ struct fsl_mc_device *mc_dev = vdev->mc_dev;
+ int ret = 0;
+
+ /* Non-dprc devices share mc_io from parent */
+ if (!is_fsl_mc_bus_dprc(mc_dev)) {
+ struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+ mc_dev->mc_io = mc_cont->mc_io;
+ return 0;
+ }
+
+ /* open DPRC, allocate a MC portal */
+ ret = dprc_setup(mc_dev);
+ if (ret < 0) {
+ dev_err(&mc_dev->dev, "Failed to setup DPRC (error = %d)\n", ret);
+ return ret;
+ }
+
+ ret = dprc_scan_container(mc_dev, mc_dev->driver_override, false);
+ if (ret < 0) {
+ dev_err(&mc_dev->dev, "Container scanning failed: %d\n", ret);
+ dprc_cleanup(mc_dev);
+ }
+
+ return 0;
+}
static const struct vfio_device_ops vfio_fsl_mc_ops = {
.name = "vfio-fsl-mc",
.open = vfio_fsl_mc_open,
@@ -112,9 +140,42 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
+ ret = vfio_fsl_mc_init_device(vdev);
+ if (ret) {
+ vfio_iommu_group_put(group, dev);
+ return ret;
+ }
+
return ret;
}
+static int vfio_fsl_mc_device_remove(struct device *dev, void *data)
+{
+ struct fsl_mc_device *mc_dev;
+
+ WARN_ON(!dev);
+ mc_dev = to_fsl_mc_device(dev);
+ if (WARN_ON(!mc_dev))
+ return -ENODEV;
+
+ kfree(mc_dev->driver_override);
+ mc_dev->driver_override = NULL;
+
+ /*
+ * The device-specific remove callback will get invoked by device_del()
+ */
+ device_del(&mc_dev->dev);
+ put_device(&mc_dev->dev);
+
+ return 0;
+}
+
+static void vfio_fsl_mc_cleanup_dprc(struct fsl_mc_device *mc_dev)
+{
+ device_for_each_child(&mc_dev->dev, NULL, vfio_fsl_mc_device_remove);
+ dprc_cleanup(mc_dev);
+}
+
static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
{
struct vfio_fsl_mc_device *vdev;
@@ -124,6 +185,11 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
+ if (is_fsl_mc_bus_dprc(mc_dev))
+ vfio_fsl_mc_cleanup_dprc(vdev->mc_dev);
+
+ mc_dev->mc_io = NULL;
+
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
devm_kfree(dev, vdev);
--
2.17.1
From: Diana Craciun <[email protected]>
Only the DPRC object allocates interrupts from the MSI
interrupt domain. The interrupts are managed by the DPRC in
a pool of interrupts. The access to this pool of interrupts
has to be protected with a lock.
This patch extends the current lock implementation to have a
lock per DPRC.
Signed-off-by: Diana Craciun <[email protected]>
---
drivers/vfio/fsl-mc/vfio_fsl_mc.c | 90 +++++++++++++++++++++--
drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 7 +-
2 files changed, 90 insertions(+), 7 deletions(-)
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 6625b7cb0a3e..ea5e81e7791c 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,76 @@
#include "vfio_fsl_mc_private.h"
+static DEFINE_MUTEX(reflck_lock);
+
+static void vfio_fsl_mc_reflck_get(struct vfio_fsl_mc_reflck *reflck)
+{
+ kref_get(&reflck->kref);
+}
+
+static void vfio_fsl_mc_reflck_release(struct kref *kref)
+{
+ struct vfio_fsl_mc_reflck *reflck = container_of(kref,
+ struct vfio_fsl_mc_reflck,
+ kref);
+
+ kfree(reflck);
+ mutex_unlock(&reflck_lock);
+}
+
+static void vfio_fsl_mc_reflck_put(struct vfio_fsl_mc_reflck *reflck)
+{
+ kref_put_mutex(&reflck->kref, vfio_fsl_mc_reflck_release, &reflck_lock);
+}
+
+static struct vfio_fsl_mc_reflck *vfio_fsl_mc_reflck_alloc(void)
+{
+ struct vfio_fsl_mc_reflck *reflck;
+
+ reflck = kzalloc(sizeof(*reflck), GFP_KERNEL);
+ if (!reflck)
+ return ERR_PTR(-ENOMEM);
+
+ kref_init(&reflck->kref);
+ mutex_init(&reflck->lock);
+
+ return reflck;
+}
+
+static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev)
+{
+ int ret = 0;
+
+ mutex_lock(&reflck_lock);
+ if (is_fsl_mc_bus_dprc(vdev->mc_dev)) {
+ vdev->reflck = vfio_fsl_mc_reflck_alloc();
+ } else {
+ struct device *mc_cont_dev = vdev->mc_dev->dev.parent;
+ struct vfio_device *device;
+ struct vfio_fsl_mc_device *cont_vdev;
+
+ device = vfio_device_get_from_dev(mc_cont_dev);
+ if (!device) {
+ ret = -ENODEV;
+ goto unlock;
+ }
+
+ cont_vdev = vfio_device_data(device);
+ if (!cont_vdev->reflck) {
+ vfio_device_put(device);
+ ret = -ENODEV;
+ goto unlock;
+ }
+ vfio_fsl_mc_reflck_get(cont_vdev->reflck);
+ vdev->reflck = cont_vdev->reflck;
+ vfio_device_put(device);
+ }
+
+unlock:
+ mutex_unlock(&reflck_lock);
+ return ret;
+}
+
static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
{
struct fsl_mc_device *mc_dev = vdev->mc_dev;
@@ -56,7 +126,7 @@ static int vfio_fsl_mc_open(void *device_data)
if (!try_module_get(THIS_MODULE))
return -ENODEV;
- mutex_lock(&vdev->driver_lock);
+ mutex_lock(&vdev->reflck->lock);
if (!vdev->refcnt) {
ret = vfio_fsl_mc_regions_init(vdev);
if (ret)
@@ -64,11 +134,11 @@ static int vfio_fsl_mc_open(void *device_data)
}
vdev->refcnt++;
- mutex_unlock(&vdev->driver_lock);
+ mutex_unlock(&vdev->reflck->lock);
return 0;
err_reg_init:
- mutex_unlock(&vdev->driver_lock);
+ mutex_unlock(&vdev->reflck->lock);
module_put(THIS_MODULE);
return ret;
}
@@ -77,12 +147,12 @@ static void vfio_fsl_mc_release(void *device_data)
{
struct vfio_fsl_mc_device *vdev = device_data;
- mutex_lock(&vdev->driver_lock);
+ mutex_lock(&vdev->reflck->lock);
if (!(--vdev->refcnt))
vfio_fsl_mc_regions_cleanup(vdev);
- mutex_unlock(&vdev->driver_lock);
+ mutex_unlock(&vdev->reflck->lock);
module_put(THIS_MODULE);
}
@@ -294,12 +364,18 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
+ ret = vfio_fsl_mc_reflck_attach(vdev);
+ if (ret) {
+ vfio_iommu_group_put(group, dev);
+ return ret;
+ }
+
ret = vfio_fsl_mc_init_device(vdev);
if (ret) {
+ vfio_fsl_mc_reflck_put(vdev->reflck);
vfio_iommu_group_put(group, dev);
return ret;
}
- mutex_init(&vdev->driver_lock);
return ret;
}
@@ -340,6 +416,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
+ vfio_fsl_mc_reflck_put(vdev->reflck);
+
if (is_fsl_mc_bus_dprc(mc_dev))
vfio_fsl_mc_cleanup_dprc(vdev->mc_dev);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index e2a0ccdd8242..d072cccd93e0 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -17,6 +17,11 @@
#define VFIO_DPRC_REGION_CACHEABLE 0x00000001
+struct vfio_fsl_mc_reflck {
+ struct kref kref;
+ struct mutex lock;
+};
+
struct vfio_fsl_mc_region {
u32 flags;
u32 type;
@@ -29,7 +34,7 @@ struct vfio_fsl_mc_device {
int refcnt;
u32 num_regions;
struct vfio_fsl_mc_region *regions;
- struct mutex driver_lock;
+ struct vfio_fsl_mc_reflck *reflck;
};
#endif /* VFIO_PCI_PRIVATE_H */
--
2.17.1
From: Diana Craciun <[email protected]>
The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).
This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.
Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.
Signed-off-by: Bharat Bhushan <[email protected]>
Signed-off-by: Diana Craciun <[email protected]>
---
drivers/vfio/fsl-mc/vfio_fsl_mc.c | 122 +++++++++++++++++++++-
drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 1 +
2 files changed, 121 insertions(+), 2 deletions(-)
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index ceb9d6b06624..107b4ab7b2d8 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
#include <linux/types.h>
#include <linux/vfio.h>
#include <linux/fsl/mc.h>
+#include <linux/delay.h>
#include "vfio_fsl_mc_private.h"
@@ -114,6 +115,11 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
{
+ int i;
+
+ for (i = 0; i < vdev->num_regions; i++)
+ iounmap(vdev->regions[i].ioaddr);
+
vdev->num_regions = 0;
kfree(vdev->regions);
}
@@ -307,13 +313,125 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,
static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
{
- return -EINVAL;
+ struct vfio_fsl_mc_device *vdev = device_data;
+ unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+ loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+ struct vfio_fsl_mc_region *region;
+ u64 data[8];
+ int i;
+
+ /* Read ioctl supported only for DPRC and DPMCP device */
+ if (strcmp(vdev->mc_dev->obj_desc.type, "dprc") &&
+ strcmp(vdev->mc_dev->obj_desc.type, "dpmcp"))
+ return -EINVAL;
+
+ if (index >= vdev->num_regions)
+ return -EINVAL;
+
+ region = &vdev->regions[index];
+
+ if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+ return -EINVAL;
+
+ if (!region->ioaddr) {
+ region->ioaddr = ioremap(region->addr, region->size);
+ if (!region->ioaddr)
+ return -ENOMEM;
+ }
+
+ if (count != 64 || off != 0)
+ return -EINVAL;
+
+ for (i = 7; i >= 0; i--)
+ data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+ if (copy_to_user(buf, data, 64))
+ return -EFAULT;
+
+ return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS 5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS 500
+
+static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data)
+{
+ int i;
+ enum mc_cmd_status status;
+ unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000;
+
+ /* Write at command parameter into portal */
+ for (i = 7; i >= 1; i--)
+ writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t));
+
+ /* Write command header in the end */
+ writeq(cmd_data[0], ioaddr);
+
+ /* Wait for response before returning to user-space
+ * This can be optimized in future to even prepare response
+ * before returning to user-space and avoid read ioctl.
+ */
+ for (;;) {
+ u64 header;
+ struct mc_cmd_header *resp_hdr;
+
+ header = cpu_to_le64(readq_relaxed(ioaddr));
+
+ resp_hdr = (struct mc_cmd_header *)&header;
+ status = (enum mc_cmd_status)resp_hdr->status;
+ if (status != MC_CMD_STATUS_READY)
+ break;
+
+ udelay(MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS);
+ timeout_usecs -= MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS;
+ if (timeout_usecs == 0)
+ return -ETIMEDOUT;
+ }
+
+ return 0;
}
static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf,
size_t count, loff_t *ppos)
{
- return -EINVAL;
+ struct vfio_fsl_mc_device *vdev = device_data;
+ unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+ loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+ struct vfio_fsl_mc_region *region;
+ u64 data[8];
+ int ret;
+
+ /* Write ioctl supported only for DPRC and DPMCP device */
+ if (strcmp(vdev->mc_dev->obj_desc.type, "dprc") &&
+ strcmp(vdev->mc_dev->obj_desc.type, "dpmcp"))
+ return -EINVAL;
+
+ if (index >= vdev->num_regions)
+ return -EINVAL;
+
+ region = &vdev->regions[index];
+
+ if (!(region->flags & VFIO_REGION_INFO_FLAG_WRITE))
+ return -EINVAL;
+
+ if (!region->ioaddr) {
+ region->ioaddr = ioremap(region->addr, region->size);
+ if (!region->ioaddr)
+ return -ENOMEM;
+ }
+
+ if (count != 64 || off != 0)
+ return -EINVAL;
+
+ if (copy_from_user(&data, buf, 64))
+ return -EFAULT;
+
+ ret = vfio_fsl_mc_send_command(region->ioaddr, data);
+ if (ret)
+ return ret;
+
+ return count;
+
}
static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region,
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index cac0b205c3d4..70824c4b10d6 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -34,6 +34,7 @@ struct vfio_fsl_mc_region {
u32 type;
u64 addr;
resource_size_t size;
+ void __iomem *ioaddr;
};
struct vfio_fsl_mc_device {
--
2.17.1
Expose to userspace information about the memory regions.
Signed-off-by: Bharat Bhushan <[email protected]>
Signed-off-by: Diana Craciun <[email protected]>
---
drivers/vfio/fsl-mc/vfio_fsl_mc.c | 77 ++++++++++++++++++++++-
drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 19 ++++++
2 files changed, 95 insertions(+), 1 deletion(-)
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index ab1cde375fc6..094df3aa3710 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,16 +15,71 @@
#include "vfio_fsl_mc_private.h"
+static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
+{
+ struct fsl_mc_device *mc_dev = vdev->mc_dev;
+ int count = mc_dev->obj_desc.region_count;
+ int i;
+
+ vdev->regions = kcalloc(count, sizeof(struct vfio_fsl_mc_region),
+ GFP_KERNEL);
+ if (!vdev->regions)
+ return -ENOMEM;
+
+ for (i = 0; i < count; i++) {
+ struct resource *res = &mc_dev->regions[i];
+
+ vdev->regions[i].addr = res->start;
+ vdev->regions[i].size = PAGE_ALIGN((resource_size(res)));
+ vdev->regions[i].flags = 0;
+ }
+
+ vdev->num_regions = mc_dev->obj_desc.region_count;
+ return 0;
+}
+
+static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
+{
+ vdev->num_regions = 0;
+ kfree(vdev->regions);
+}
+
static int vfio_fsl_mc_open(void *device_data)
{
+ struct vfio_fsl_mc_device *vdev = device_data;
+ int ret;
+
if (!try_module_get(THIS_MODULE))
return -ENODEV;
+ mutex_lock(&vdev->driver_lock);
+ if (!vdev->refcnt) {
+ ret = vfio_fsl_mc_regions_init(vdev);
+ if (ret)
+ goto err_reg_init;
+ }
+ vdev->refcnt++;
+
+ mutex_unlock(&vdev->driver_lock);
return 0;
+
+err_reg_init:
+ mutex_unlock(&vdev->driver_lock);
+ module_put(THIS_MODULE);
+ return ret;
}
static void vfio_fsl_mc_release(void *device_data)
{
+ struct vfio_fsl_mc_device *vdev = device_data;
+
+ mutex_lock(&vdev->driver_lock);
+
+ if (!(--vdev->refcnt))
+ vfio_fsl_mc_regions_cleanup(vdev);
+
+ mutex_unlock(&vdev->driver_lock);
+
module_put(THIS_MODULE);
}
@@ -58,7 +113,26 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,
}
case VFIO_DEVICE_GET_REGION_INFO:
{
- return -EINVAL;
+ struct vfio_region_info info;
+
+ minsz = offsetofend(struct vfio_region_info, offset);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz)
+ return -EINVAL;
+
+ if (info.index >= vdev->num_regions)
+ return -EINVAL;
+
+ /* map offset to the physical address */
+ info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
+ info.size = vdev->regions[info.index].size;
+ info.flags = vdev->regions[info.index].flags;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -165,6 +239,7 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
vfio_iommu_group_put(group, dev);
return ret;
}
+ mutex_init(&vdev->driver_lock);
return ret;
}
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index b92858a003c0..764e2cf2c70d 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -7,8 +7,27 @@
#ifndef VFIO_FSL_MC_PRIVATE_H
#define VFIO_FSL_MC_PRIVATE_H
+#define VFIO_FSL_MC_OFFSET_SHIFT 40
+#define VFIO_FSL_MC_OFFSET_MASK (((u64)(1) << VFIO_FSL_MC_OFFSET_SHIFT) - 1)
+
+#define VFIO_FSL_MC_OFFSET_TO_INDEX(off) ((off) >> VFIO_FSL_MC_OFFSET_SHIFT)
+
+#define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \
+ ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT)
+
+struct vfio_fsl_mc_region {
+ u32 flags;
+ u32 type;
+ u64 addr;
+ resource_size_t size;
+};
+
struct vfio_fsl_mc_device {
struct fsl_mc_device *mc_dev;
+ int refcnt;
+ u32 num_regions;
+ struct vfio_fsl_mc_region *regions;
+ struct mutex driver_lock;
};
#endif /* VFIO_PCI_PRIVATE_H */
--
2.17.1
On Mon, 23 Mar 2020 19:19:02 +0200
Diana Craciun <[email protected]> wrote:
> DPAA2 (Data Path Acceleration Architecture) consists in
> mechanisms for processing Ethernet packets, queue management,
> accelerators, etc.
>
> The Management Complex (mc) is a hardware entity that manages the DPAA2
> hardware resources. It provides an object-based abstraction for software
> drivers to use the DPAA2 hardware. The MC mediates operations such as
> create, discover, destroy of DPAA2 objects.
> The MC provides memory-mapped I/O command interfaces (MC portals) which
> DPAA2 software drivers use to operate on DPAA2 objects.
>
> A DPRC is a container object that holds other types of DPAA2 objects.
> Each object in the DPRC is a Linux device and bound to a driver.
> The MC-bus driver is a platform driver (different from PCI or platform
> bus). The DPRC driver does runtime management of a bus instance. It
> performs the initial scan of the DPRC and handles changes in the DPRC
> configuration (adding/removing objects).
>
> All objects inside a container share the same hardware isolation
> context, meaning that only an entire DPRC can be assigned to
> a virtual machine.
> When a container is assigned to a virtual machine, all the objects
> within that container are assigned to that virtual machine.
> The DPRC container assigned to the virtual machine is not allowed
> to change contents (add/remove objects) by the guest. The restriction
> is set by the host and enforced by the mc hardware.
>
> The DPAA2 objects can be directly assigned to the guest. However
> the MC portals (the memory mapped command interface to the MC) need
> to be emulated because there are commands that configure the
> interrupts and the isolation IDs which are virtual in the guest.
>
> Example:
> echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
> echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
>
> The dprc.2 is bound to the VFIO driver and all the objects within
> dprc.2 are going to be bound to the VFIO driver.
What's the composition of the IOMMU group, does it start with the DPRC
and each of the objects within the container are added to the same
group as they're created?
For an alternative to the driver_override mechanism used in this series
of passing the override through various scan/create callbacks, you
might consider something like I did for PCI SR-IOV:
https://lore.kernel.org/lkml/[email protected]/
ie. using the bus notifier to setup the driver_override before driver
matching is done. Thanks,
Alex
> More details about the DPAA2 objects can be found here:
> Documentation/networking/device_drivers/freescale/dpaa2/overview.rst
>
> The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
> driver. The changes were needed in order to re-use code and to export
> some more functions that are needed by the VFIO driver.
> Currenlty the mc-bus patches are under review:
> https://www.spinics.net/lists/kernel/msg3447567.html
>
> Bharat Bhushan (1):
> vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices
>
> Diana Craciun (8):
> vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
> vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
> vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
> vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
> vfio/fsl-mc: Added lock support in preparation for interrupt handling
> vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
> vfio/fsl-mc: trigger an interrupt via eventfd
> vfio/fsl-mc: Add read/write support for fsl-mc devices
>
> MAINTAINERS | 6 +
> drivers/vfio/Kconfig | 1 +
> drivers/vfio/Makefile | 1 +
> drivers/vfio/fsl-mc/Kconfig | 9 +
> drivers/vfio/fsl-mc/Makefile | 4 +
> drivers/vfio/fsl-mc/vfio_fsl_mc.c | 660 ++++++++++++++++++++++
> drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 221 ++++++++
> drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 56 ++
> include/uapi/linux/vfio.h | 1 +
> 9 files changed, 959 insertions(+)
> create mode 100644 drivers/vfio/fsl-mc/Kconfig
> create mode 100644 drivers/vfio/fsl-mc/Makefile
> create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
> create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
> create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
>
Thanks for looking into this.
On 3/27/2020 11:11 PM, Alex Williamson wrote:
> On Mon, 23 Mar 2020 19:19:02 +0200
> Diana Craciun <[email protected]> wrote:
>
>> DPAA2 (Data Path Acceleration Architecture) consists in
>> mechanisms for processing Ethernet packets, queue management,
>> accelerators, etc.
>>
>> The Management Complex (mc) is a hardware entity that manages the DPAA2
>> hardware resources. It provides an object-based abstraction for software
>> drivers to use the DPAA2 hardware. The MC mediates operations such as
>> create, discover, destroy of DPAA2 objects.
>> The MC provides memory-mapped I/O command interfaces (MC portals) which
>> DPAA2 software drivers use to operate on DPAA2 objects.
>>
>> A DPRC is a container object that holds other types of DPAA2 objects.
>> Each object in the DPRC is a Linux device and bound to a driver.
>> The MC-bus driver is a platform driver (different from PCI or platform
>> bus). The DPRC driver does runtime management of a bus instance. It
>> performs the initial scan of the DPRC and handles changes in the DPRC
>> configuration (adding/removing objects).
>>
>> All objects inside a container share the same hardware isolation
>> context, meaning that only an entire DPRC can be assigned to
>> a virtual machine.
>> When a container is assigned to a virtual machine, all the objects
>> within that container are assigned to that virtual machine.
>> The DPRC container assigned to the virtual machine is not allowed
>> to change contents (add/remove objects) by the guest. The restriction
>> is set by the host and enforced by the mc hardware.
>>
>> The DPAA2 objects can be directly assigned to the guest. However
>> the MC portals (the memory mapped command interface to the MC) need
>> to be emulated because there are commands that configure the
>> interrupts and the isolation IDs which are virtual in the guest.
>>
>> Example:
>> echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
>> echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind
>>
>> The dprc.2 is bound to the VFIO driver and all the objects within
>> dprc.2 are going to be bound to the VFIO driver.
> What's the composition of the IOMMU group, does it start with the DPRC
> and each of the objects within the container are added to the same
> group as they're created?
Yes, the IOMMU group starts with the DPRC and the other objects are then
added to that group.
>
> For an alternative to the driver_override mechanism used in this series
> of passing the override through various scan/create callbacks, you
> might consider something like I did for PCI SR-IOV:
>
> https://lore.kernel.org/lkml/[email protected]/
>
> ie. using the bus notifier to setup the driver_override before driver
> matching is done. Thanks,
Thanks, I like your approach. I will give it a try.
Diana
>
> Alex
>
>> More details about the DPAA2 objects can be found here:
>> Documentation/networking/device_drivers/freescale/dpaa2/overview.rst
>>
>> The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
>> driver. The changes were needed in order to re-use code and to export
>> some more functions that are needed by the VFIO driver.
>> Currenlty the mc-bus patches are under review:
>> https://www.spinics.net/lists/kernel/msg3447567.html
>>
>> Bharat Bhushan (1):
>> vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices
>>
>> Diana Craciun (8):
>> vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
>> vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
>> vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
>> vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
>> vfio/fsl-mc: Added lock support in preparation for interrupt handling
>> vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
>> vfio/fsl-mc: trigger an interrupt via eventfd
>> vfio/fsl-mc: Add read/write support for fsl-mc devices
>>
>> MAINTAINERS | 6 +
>> drivers/vfio/Kconfig | 1 +
>> drivers/vfio/Makefile | 1 +
>> drivers/vfio/fsl-mc/Kconfig | 9 +
>> drivers/vfio/fsl-mc/Makefile | 4 +
>> drivers/vfio/fsl-mc/vfio_fsl_mc.c | 660 ++++++++++++++++++++++
>> drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 221 ++++++++
>> drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 56 ++
>> include/uapi/linux/vfio.h | 1 +
>> 9 files changed, 959 insertions(+)
>> create mode 100644 drivers/vfio/fsl-mc/Kconfig
>> create mode 100644 drivers/vfio/fsl-mc/Makefile
>> create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
>> create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
>> create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
>>