Hi,
This series attempts to add vfio live migration support for HiSilicon
ACC VF devices based on the new v2 migration protocol definition and
mlx5 v9 series discussed here[0].
v8 --> v9
- Added acks by Wangzhou/Longfang/Yekai
- Added R-by tags by Jason.
- Addressed comments by Alex on v8.
- Fixed the pf_queue pointer assignment error in patch #8.
- Addressed comments from Kevin,
-Updated patch #5 commit log msg with a clarification that VF
migration BAR assignment is fine if migration support is not there.
-Added QM description to patch #8 commit msg.
This is sanity tested on a HiSilicon platform using the Qemu branch
provided here[1].
Please take a look and let me know your feedback.
Thanks,
Shameer
[0] https://lore.kernel.org/kvm/[email protected]/
[1] https://github.com/jgunthorpe/qemu/commits/vfio_migration_v2
v7 --> v8
- Dropped PRE_COPY support and early compatibility checking based on
the discussion here[1].
- Addressed comments from John, Jason & Alex (Thanks!).
v6 --> v7
-Renamed MIG_PRECOPY ioctl name and struct name. Updated ioctl descriptions
regarding ioctl validity (patch #7).
- Adressed comments from Jason and Alex on PRE_COPY read() and ioctl() fns
(patch #9).
- Moved only VF PCI ids to pci_ids.h(patch #3).
v5 --> v6
-Report PRE_COPY support and use that for early compatibility check
between src and dst devices.
-For generic PRE_COPY support, included patch #7 from Jason(Thanks!).
-Addressed comments from Alex(Thanks!).
-Added the QM state register update to QM driver(patch #8) since that
is being used in migration driver to decide whether the device is
ready to save the state.
RFCv4 --> v5
- Dropped RFC tag as v2 migration APIs are more stable now.
- Addressed review comments from Jason and Alex (Thanks!).
v3 --> RFCv4
-Based on migration v2 protocol and mlx5 v7 series.
-Added RFC tag again as migration v2 protocol is still under discussion.
-Added new patch #6 to retrieve the PF QM data.
-PRE_COPY compatibility check is now done after the migration data
transfer. This is not ideal and needs discussion.
RFC v2 --> v3
-Dropped RFC tag as the vfio_pci_core subsystem framework is now
part of 5.15-rc1.
-Added override methods for vfio_device_ops read/write/mmap calls
to limit the access within the functional register space.
-Patches 1 to 3 are code refactoring to move the common ACC QM
definitions and header around.
RFCv1 --> RFCv2
-Adds a new vendor-specific vfio_pci driver(hisi-acc-vfio-pci)
for HiSilicon ACC VF devices based on the new vfio-pci-core
framework proposal.
-Since HiSilicon ACC VF device MMIO space contains both the
functional register space and migration control register space,
override the vfio_device_ops ioctl method to report only the
functional space to VMs.
-For a successful migration, we still need access to VF dev
functional register space mainly to read the status registers.
But accessing these while the Guest vCPUs are running may leave
a security hole. To avoid any potential security issues, we
map/unmap the MMIO regions on a need basis and is safe to do so.
(Please see hisi_acc_vf_ioremap/unmap() fns in patch #4).
-Dropped debugfs support for now.
-Uses common QM functions for mailbox access(patch #3).
Longfang Liu (3):
crypto: hisilicon/qm: Move few definitions to common header
crypto: hisilicon/qm: Set the VF QM state register
hisi_acc_vfio_pci: Add support for VFIO live migration
Shameer Kolothum (6):
crypto: hisilicon/qm: Move the QM header to include/linux
hisi_acc_qm: Move VF PCI device IDs to common header
hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices
hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
hisi_acc_vfio_pci: Use its own PCI reset_done error handler
MAINTAINERS | 7 +
drivers/crypto/hisilicon/hpre/hpre.h | 2 +-
drivers/crypto/hisilicon/hpre/hpre_main.c | 19 +-
drivers/crypto/hisilicon/qm.c | 68 +-
drivers/crypto/hisilicon/sec2/sec.h | 2 +-
drivers/crypto/hisilicon/sec2/sec_main.c | 21 +-
drivers/crypto/hisilicon/sgl.c | 2 +-
drivers/crypto/hisilicon/zip/zip.h | 2 +-
drivers/crypto/hisilicon/zip/zip_main.c | 17 +-
drivers/vfio/pci/Kconfig | 2 +
drivers/vfio/pci/Makefile | 2 +
drivers/vfio/pci/hisilicon/Kconfig | 15 +
drivers/vfio/pci/hisilicon/Makefile | 4 +
.../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 1326 +++++++++++++++++
.../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 116 ++
.../qm.h => include/linux/hisi_acc_qm.h | 49 +
include/linux/pci_ids.h | 3 +
17 files changed, 1591 insertions(+), 66 deletions(-)
create mode 100644 drivers/vfio/pci/hisilicon/Kconfig
create mode 100644 drivers/vfio/pci/hisilicon/Makefile
create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (87%)
--
2.25.1
From: Longfang Liu <[email protected]>
Move Doorbell and Mailbox definitions to common header file.
Also export QM mailbox functions.
This will be useful when we introduce VFIO PCI HiSilicon ACC live
migration driver.
Signed-off-by: Longfang Liu <[email protected]>
Acked-by: Zhou Wang <[email protected]>
Signed-off-by: Shameer Kolothum <[email protected]>
---
drivers/crypto/hisilicon/qm.c | 58 +++++++++++------------------------
include/linux/hisi_acc_qm.h | 38 +++++++++++++++++++++++
2 files changed, 56 insertions(+), 40 deletions(-)
diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index ed23e1d3fa27..c88e013371af 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -33,23 +33,6 @@
#define QM_ABNORMAL_EVENT_IRQ_VECTOR 3
/* mailbox */
-#define QM_MB_CMD_SQC 0x0
-#define QM_MB_CMD_CQC 0x1
-#define QM_MB_CMD_EQC 0x2
-#define QM_MB_CMD_AEQC 0x3
-#define QM_MB_CMD_SQC_BT 0x4
-#define QM_MB_CMD_CQC_BT 0x5
-#define QM_MB_CMD_SQC_VFT_V2 0x6
-#define QM_MB_CMD_STOP_QP 0x8
-#define QM_MB_CMD_SRC 0xc
-#define QM_MB_CMD_DST 0xd
-
-#define QM_MB_CMD_SEND_BASE 0x300
-#define QM_MB_EVENT_SHIFT 8
-#define QM_MB_BUSY_SHIFT 13
-#define QM_MB_OP_SHIFT 14
-#define QM_MB_CMD_DATA_ADDR_L 0x304
-#define QM_MB_CMD_DATA_ADDR_H 0x308
#define QM_MB_PING_ALL_VFS 0xffff
#define QM_MB_CMD_DATA_SHIFT 32
#define QM_MB_CMD_DATA_MASK GENMASK(31, 0)
@@ -103,19 +86,12 @@
#define QM_DB_CMD_SHIFT_V1 16
#define QM_DB_INDEX_SHIFT_V1 32
#define QM_DB_PRIORITY_SHIFT_V1 48
-#define QM_DOORBELL_SQ_CQ_BASE_V2 0x1000
-#define QM_DOORBELL_EQ_AEQ_BASE_V2 0x2000
#define QM_QUE_ISO_CFG_V 0x0030
#define QM_PAGE_SIZE 0x0034
#define QM_QUE_ISO_EN 0x100154
#define QM_CAPBILITY 0x100158
#define QM_QP_NUN_MASK GENMASK(10, 0)
#define QM_QP_DB_INTERVAL 0x10000
-#define QM_QP_MAX_NUM_SHIFT 11
-#define QM_DB_CMD_SHIFT_V2 12
-#define QM_DB_RAND_SHIFT_V2 16
-#define QM_DB_INDEX_SHIFT_V2 32
-#define QM_DB_PRIORITY_SHIFT_V2 48
#define QM_MEM_START_INIT 0x100040
#define QM_MEM_INIT_DONE 0x100044
@@ -693,7 +669,7 @@ static void qm_mb_pre_init(struct qm_mailbox *mailbox, u8 cmd,
}
/* return 0 mailbox ready, -ETIMEDOUT hardware timeout */
-static int qm_wait_mb_ready(struct hisi_qm *qm)
+int hisi_qm_wait_mb_ready(struct hisi_qm *qm)
{
u32 val;
@@ -701,6 +677,7 @@ static int qm_wait_mb_ready(struct hisi_qm *qm)
val, !((val >> QM_MB_BUSY_SHIFT) &
0x1), POLL_PERIOD, POLL_TIMEOUT);
}
+EXPORT_SYMBOL_GPL(hisi_qm_wait_mb_ready);
/* 128 bit should be written to hardware at one time to trigger a mailbox */
static void qm_mb_write(struct hisi_qm *qm, const void *src)
@@ -726,14 +703,14 @@ static void qm_mb_write(struct hisi_qm *qm, const void *src)
static int qm_mb_nolock(struct hisi_qm *qm, struct qm_mailbox *mailbox)
{
- if (unlikely(qm_wait_mb_ready(qm))) {
+ if (unlikely(hisi_qm_wait_mb_ready(qm))) {
dev_err(&qm->pdev->dev, "QM mailbox is busy to start!\n");
goto mb_busy;
}
qm_mb_write(qm, mailbox);
- if (unlikely(qm_wait_mb_ready(qm))) {
+ if (unlikely(hisi_qm_wait_mb_ready(qm))) {
dev_err(&qm->pdev->dev, "QM mailbox operation timeout!\n");
goto mb_busy;
}
@@ -745,8 +722,8 @@ static int qm_mb_nolock(struct hisi_qm *qm, struct qm_mailbox *mailbox)
return -EBUSY;
}
-static int qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
- bool op)
+int hisi_qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
+ bool op)
{
struct qm_mailbox mailbox;
int ret;
@@ -762,6 +739,7 @@ static int qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
return ret;
}
+EXPORT_SYMBOL_GPL(hisi_qm_mb);
static void qm_db_v1(struct hisi_qm *qm, u16 qn, u8 cmd, u16 index, u8 priority)
{
@@ -1351,7 +1329,7 @@ static int qm_get_vft_v2(struct hisi_qm *qm, u32 *base, u32 *number)
u64 sqc_vft;
int ret;
- ret = qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_VFT_V2, 0, 0, 1);
if (ret)
return ret;
@@ -1725,12 +1703,12 @@ static int dump_show(struct hisi_qm *qm, void *info,
static int qm_dump_sqc_raw(struct hisi_qm *qm, dma_addr_t dma_addr, u16 qp_id)
{
- return qm_mb(qm, QM_MB_CMD_SQC, dma_addr, qp_id, 1);
+ return hisi_qm_mb(qm, QM_MB_CMD_SQC, dma_addr, qp_id, 1);
}
static int qm_dump_cqc_raw(struct hisi_qm *qm, dma_addr_t dma_addr, u16 qp_id)
{
- return qm_mb(qm, QM_MB_CMD_CQC, dma_addr, qp_id, 1);
+ return hisi_qm_mb(qm, QM_MB_CMD_CQC, dma_addr, qp_id, 1);
}
static int qm_sqc_dump(struct hisi_qm *qm, const char *s)
@@ -1842,7 +1820,7 @@ static int qm_eqc_aeqc_dump(struct hisi_qm *qm, char *s, size_t size,
if (IS_ERR(xeqc))
return PTR_ERR(xeqc);
- ret = qm_mb(qm, cmd, xeqc_dma, 0, 1);
+ ret = hisi_qm_mb(qm, cmd, xeqc_dma, 0, 1);
if (ret)
goto err_free_ctx;
@@ -2495,7 +2473,7 @@ static int qm_ping_pf(struct hisi_qm *qm, u64 cmd)
static int qm_stop_qp(struct hisi_qp *qp)
{
- return qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
+ return hisi_qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
}
static int qm_set_msi(struct hisi_qm *qm, bool set)
@@ -2763,7 +2741,7 @@ static int qm_sq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid)
return -ENOMEM;
}
- ret = qm_mb(qm, QM_MB_CMD_SQC, sqc_dma, qp_id, 0);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_SQC, sqc_dma, qp_id, 0);
dma_unmap_single(dev, sqc_dma, sizeof(struct qm_sqc), DMA_TO_DEVICE);
kfree(sqc);
@@ -2804,7 +2782,7 @@ static int qm_cq_ctx_cfg(struct hisi_qp *qp, int qp_id, u32 pasid)
return -ENOMEM;
}
- ret = qm_mb(qm, QM_MB_CMD_CQC, cqc_dma, qp_id, 0);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_CQC, cqc_dma, qp_id, 0);
dma_unmap_single(dev, cqc_dma, sizeof(struct qm_cqc), DMA_TO_DEVICE);
kfree(cqc);
@@ -3655,7 +3633,7 @@ static int qm_eq_ctx_cfg(struct hisi_qm *qm)
return -ENOMEM;
}
- ret = qm_mb(qm, QM_MB_CMD_EQC, eqc_dma, 0, 0);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_EQC, eqc_dma, 0, 0);
dma_unmap_single(dev, eqc_dma, sizeof(struct qm_eqc), DMA_TO_DEVICE);
kfree(eqc);
@@ -3684,7 +3662,7 @@ static int qm_aeq_ctx_cfg(struct hisi_qm *qm)
return -ENOMEM;
}
- ret = qm_mb(qm, QM_MB_CMD_AEQC, aeqc_dma, 0, 0);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_AEQC, aeqc_dma, 0, 0);
dma_unmap_single(dev, aeqc_dma, sizeof(struct qm_aeqc), DMA_TO_DEVICE);
kfree(aeqc);
@@ -3723,11 +3701,11 @@ static int __hisi_qm_start(struct hisi_qm *qm)
if (ret)
return ret;
- ret = qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0);
if (ret)
return ret;
- ret = qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
+ ret = hisi_qm_mb(qm, QM_MB_CMD_CQC_BT, qm->cqc_dma, 0, 0);
if (ret)
return ret;
diff --git a/include/linux/hisi_acc_qm.h b/include/linux/hisi_acc_qm.h
index 3068093229a5..6a6477c34666 100644
--- a/include/linux/hisi_acc_qm.h
+++ b/include/linux/hisi_acc_qm.h
@@ -34,6 +34,40 @@
#define QM_WUSER_M_CFG_ENABLE 0x1000a8
#define WUSER_M_CFG_ENABLE 0xffffffff
+/* mailbox */
+#define QM_MB_CMD_SQC 0x0
+#define QM_MB_CMD_CQC 0x1
+#define QM_MB_CMD_EQC 0x2
+#define QM_MB_CMD_AEQC 0x3
+#define QM_MB_CMD_SQC_BT 0x4
+#define QM_MB_CMD_CQC_BT 0x5
+#define QM_MB_CMD_SQC_VFT_V2 0x6
+#define QM_MB_CMD_STOP_QP 0x8
+#define QM_MB_CMD_SRC 0xc
+#define QM_MB_CMD_DST 0xd
+
+#define QM_MB_CMD_SEND_BASE 0x300
+#define QM_MB_EVENT_SHIFT 8
+#define QM_MB_BUSY_SHIFT 13
+#define QM_MB_OP_SHIFT 14
+#define QM_MB_CMD_DATA_ADDR_L 0x304
+#define QM_MB_CMD_DATA_ADDR_H 0x308
+#define QM_MB_MAX_WAIT_CNT 6000
+
+/* doorbell */
+#define QM_DOORBELL_CMD_SQ 0
+#define QM_DOORBELL_CMD_CQ 1
+#define QM_DOORBELL_CMD_EQ 2
+#define QM_DOORBELL_CMD_AEQ 3
+
+#define QM_DOORBELL_SQ_CQ_BASE_V2 0x1000
+#define QM_DOORBELL_EQ_AEQ_BASE_V2 0x2000
+#define QM_QP_MAX_NUM_SHIFT 11
+#define QM_DB_CMD_SHIFT_V2 12
+#define QM_DB_RAND_SHIFT_V2 16
+#define QM_DB_INDEX_SHIFT_V2 32
+#define QM_DB_PRIORITY_SHIFT_V2 48
+
/* qm cache */
#define QM_CACHE_CTL 0x100050
#define SQC_CACHE_ENABLE BIT(0)
@@ -414,6 +448,10 @@ pci_ers_result_t hisi_qm_dev_slot_reset(struct pci_dev *pdev);
void hisi_qm_reset_prepare(struct pci_dev *pdev);
void hisi_qm_reset_done(struct pci_dev *pdev);
+int hisi_qm_wait_mb_ready(struct hisi_qm *qm);
+int hisi_qm_mb(struct hisi_qm *qm, u8 cmd, dma_addr_t dma_addr, u16 queue,
+ bool op);
+
struct hisi_acc_sgl_pool;
struct hisi_acc_hw_sgl *hisi_acc_sg_buf_map_to_hw_sgl(struct device *dev,
struct scatterlist *sgl, struct hisi_acc_sgl_pool *pool,
--
2.25.1
Register private handler for pci_error_handlers.reset_done and update
state accordingly.
Reviewed-by: Jason Gunthorpe <[email protected]>
Reviewed-by: Longfang Liu <[email protected]>
Signed-off-by: Shameer Kolothum <[email protected]>
---
.../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 57 ++++++++++++++++++-
.../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 4 +-
2 files changed, 57 insertions(+), 4 deletions(-)
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 51b814f4303b..767b5d47631a 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -626,6 +626,27 @@ static void hisi_acc_vf_disable_fds(struct hisi_acc_vf_core_device *hisi_acc_vde
}
}
+/*
+ * This function is called in all state_mutex unlock cases to
+ * handle a 'deferred_reset' if exists.
+ */
+static void
+hisi_acc_vf_state_mutex_unlock(struct hisi_acc_vf_core_device *hisi_acc_vdev)
+{
+again:
+ spin_lock(&hisi_acc_vdev->reset_lock);
+ if (hisi_acc_vdev->deferred_reset) {
+ hisi_acc_vdev->deferred_reset = false;
+ spin_unlock(&hisi_acc_vdev->reset_lock);
+ hisi_acc_vdev->vf_qm_state = QM_NOT_READY;
+ hisi_acc_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
+ hisi_acc_vf_disable_fds(hisi_acc_vdev);
+ goto again;
+ }
+ mutex_unlock(&hisi_acc_vdev->state_mutex);
+ spin_unlock(&hisi_acc_vdev->reset_lock);
+}
+
static void hisi_acc_vf_start_device(struct hisi_acc_vf_core_device *hisi_acc_vdev)
{
struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
@@ -922,7 +943,7 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
break;
}
}
- mutex_unlock(&hisi_acc_vdev->state_mutex);
+ hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
return res;
}
@@ -935,10 +956,35 @@ hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
mutex_lock(&hisi_acc_vdev->state_mutex);
*curr_state = hisi_acc_vdev->mig_state;
- mutex_unlock(&hisi_acc_vdev->state_mutex);
+ hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
return 0;
}
+static void hisi_acc_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+ struct hisi_acc_vf_core_device *hisi_acc_vdev = dev_get_drvdata(&pdev->dev);
+
+ if (hisi_acc_vdev->core_device.vdev.migration_flags !=
+ VFIO_MIGRATION_STOP_COPY)
+ return;
+
+ /*
+ * As the higher VFIO layers are holding locks across reset and using
+ * those same locks with the mm_lock we need to prevent ABBA deadlock
+ * with the state_mutex and mm_lock.
+ * In case the state_mutex was taken already we defer the cleanup work
+ * to the unlock flow of the other running context.
+ */
+ spin_lock(&hisi_acc_vdev->reset_lock);
+ hisi_acc_vdev->deferred_reset = true;
+ if (!mutex_trylock(&hisi_acc_vdev->state_mutex)) {
+ spin_unlock(&hisi_acc_vdev->reset_lock);
+ return;
+ }
+ spin_unlock(&hisi_acc_vdev->reset_lock);
+ hisi_acc_vf_state_mutex_unlock(hisi_acc_vdev);
+}
+
static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev)
{
struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
@@ -1259,12 +1305,17 @@ static const struct pci_device_id hisi_acc_vfio_pci_table[] = {
MODULE_DEVICE_TABLE(pci, hisi_acc_vfio_pci_table);
+static const struct pci_error_handlers hisi_acc_vf_err_handlers = {
+ .reset_done = hisi_acc_vf_pci_aer_reset_done,
+ .error_detected = vfio_pci_core_aer_err_detected,
+};
+
static struct pci_driver hisi_acc_vfio_pci_driver = {
.name = KBUILD_MODNAME,
.id_table = hisi_acc_vfio_pci_table,
.probe = hisi_acc_vfio_pci_probe,
.remove = hisi_acc_vfio_pci_remove,
- .err_handler = &vfio_pci_core_err_handlers,
+ .err_handler = &hisi_acc_vf_err_handlers,
};
module_pci_driver(hisi_acc_vfio_pci_driver);
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
index 1c7d75408790..5494f4983bbe 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
@@ -98,6 +98,7 @@ struct hisi_acc_vf_migration_file {
struct hisi_acc_vf_core_device {
struct vfio_pci_core_device core_device;
+ u8 deferred_reset:1;
/* for migration state */
struct mutex state_mutex;
enum vfio_device_mig_state mig_state;
@@ -107,7 +108,8 @@ struct hisi_acc_vf_core_device {
struct hisi_qm vf_qm;
u32 vf_qm_state;
int vf_id;
-
+ /* for reset handler */
+ spinlock_t reset_lock;
struct hisi_acc_vf_migration_file *resuming_migf;
struct hisi_acc_vf_migration_file *saving_migf;
};
--
2.25.1
Hi Alex,
> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: 08 March 2022 18:49
> To: [email protected]; [email protected];
> [email protected]
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; Linuxarm <[email protected]>; liulongfang
> <[email protected]>; Zengtao (B) <[email protected]>;
> Jonathan Cameron <[email protected]>; Wangzhou (B)
> <[email protected]>
> Subject: [PATCH v9 0/9] vfio/hisilicon: add ACC live migration driver
>
> Hi,
>
> This series attempts to add vfio live migration support for HiSilicon
> ACC VF devices based on the new v2 migration protocol definition and
> mlx5 v9 series discussed here[0].
>
> v8 --> v9
> - Added acks by Wangzhou/Longfang/Yekai
> - Added R-by tags by Jason.
> - Addressed comments by Alex on v8.
> - Fixed the pf_queue pointer assignment error in patch #8.
> - Addressed comments from Kevin,
> -Updated patch #5 commit log msg with a clarification that VF
> migration BAR assignment is fine if migration support is not there.
> -Added QM description to patch #8 commit msg.
Hope there is nothing pending for this series now to make it to next.
I know ack from Bjorn is still pending for patch #3, and I have
sent a ping last week and also CCd him on that patch.
Please let me know if there is anything I missed.
Thanks,
Shameer
>
> This is sanity tested on a HiSilicon platform using the Qemu branch
> provided here[1].
>
> Please take a look and let me know your feedback.
>
> Thanks,
> Shameer
> [0]
> https://lore.kernel.org/kvm/[email protected]/
> [1] https://github.com/jgunthorpe/qemu/commits/vfio_migration_v2
>
> v7 --> v8
> - Dropped PRE_COPY support and early compatibility checking based on
> the discussion here[1].
> - Addressed comments from John, Jason & Alex (Thanks!).
>
> v6 --> v7
> -Renamed MIG_PRECOPY ioctl name and struct name. Updated ioctl
> descriptions
> regarding ioctl validity (patch #7).
> - Adressed comments from Jason and Alex on PRE_COPY read() and ioctl() fns
> (patch #9).
> - Moved only VF PCI ids to pci_ids.h(patch #3).
>
> v5 --> v6
> -Report PRE_COPY support and use that for early compatibility check
> between src and dst devices.
> -For generic PRE_COPY support, included patch #7 from Jason(Thanks!).
> -Addressed comments from Alex(Thanks!).
> -Added the QM state register update to QM driver(patch #8) since that
> is being used in migration driver to decide whether the device is
> ready to save the state.
>
> RFCv4 --> v5
> - Dropped RFC tag as v2 migration APIs are more stable now.
> - Addressed review comments from Jason and Alex (Thanks!).
>
> v3 --> RFCv4
> -Based on migration v2 protocol and mlx5 v7 series.
> -Added RFC tag again as migration v2 protocol is still under discussion.
> -Added new patch #6 to retrieve the PF QM data.
> -PRE_COPY compatibility check is now done after the migration data
> transfer. This is not ideal and needs discussion.
>
> RFC v2 --> v3
> -Dropped RFC tag as the vfio_pci_core subsystem framework is now
> part of 5.15-rc1.
> -Added override methods for vfio_device_ops read/write/mmap calls
> to limit the access within the functional register space.
> -Patches 1 to 3 are code refactoring to move the common ACC QM
> definitions and header around.
>
> RFCv1 --> RFCv2
>
> -Adds a new vendor-specific vfio_pci driver(hisi-acc-vfio-pci)
> for HiSilicon ACC VF devices based on the new vfio-pci-core
> framework proposal.
>
> -Since HiSilicon ACC VF device MMIO space contains both the
> functional register space and migration control register space,
> override the vfio_device_ops ioctl method to report only the
> functional space to VMs.
>
> -For a successful migration, we still need access to VF dev
> functional register space mainly to read the status registers.
> But accessing these while the Guest vCPUs are running may leave
> a security hole. To avoid any potential security issues, we
> map/unmap the MMIO regions on a need basis and is safe to do so.
> (Please see hisi_acc_vf_ioremap/unmap() fns in patch #4).
>
> -Dropped debugfs support for now.
> -Uses common QM functions for mailbox access(patch #3).
>
> Longfang Liu (3):
> crypto: hisilicon/qm: Move few definitions to common header
> crypto: hisilicon/qm: Set the VF QM state register
> hisi_acc_vfio_pci: Add support for VFIO live migration
>
> Shameer Kolothum (6):
> crypto: hisilicon/qm: Move the QM header to include/linux
> hisi_acc_qm: Move VF PCI device IDs to common header
> hisi_acc_vfio_pci: add new vfio_pci driver for HiSilicon ACC devices
> hisi_acc_vfio_pci: Restrict access to VF dev BAR2 migration region
> hisi_acc_vfio_pci: Add helper to retrieve the struct pci_driver
> hisi_acc_vfio_pci: Use its own PCI reset_done error handler
>
> MAINTAINERS | 7 +
> drivers/crypto/hisilicon/hpre/hpre.h | 2 +-
> drivers/crypto/hisilicon/hpre/hpre_main.c | 19 +-
> drivers/crypto/hisilicon/qm.c | 68 +-
> drivers/crypto/hisilicon/sec2/sec.h | 2 +-
> drivers/crypto/hisilicon/sec2/sec_main.c | 21 +-
> drivers/crypto/hisilicon/sgl.c | 2 +-
> drivers/crypto/hisilicon/zip/zip.h | 2 +-
> drivers/crypto/hisilicon/zip/zip_main.c | 17 +-
> drivers/vfio/pci/Kconfig | 2 +
> drivers/vfio/pci/Makefile | 2 +
> drivers/vfio/pci/hisilicon/Kconfig | 15 +
> drivers/vfio/pci/hisilicon/Makefile | 4 +
> .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 1326 +++++++++++++++++
> .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 116 ++
> .../qm.h => include/linux/hisi_acc_qm.h | 49 +
> include/linux/pci_ids.h | 3 +
> 17 files changed, 1591 insertions(+), 66 deletions(-)
> create mode 100644 drivers/vfio/pci/hisilicon/Kconfig
> create mode 100644 drivers/vfio/pci/hisilicon/Makefile
> create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
> create mode 100644 drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h
> rename drivers/crypto/hisilicon/qm.h => include/linux/hisi_acc_qm.h (87%)
>
> --
> 2.25.1