Hi Joerg and all,
IDXD kernel work queues were disabled due to the flawed use of kernel VA
and SVA API.
Link: https://lore.kernel.org/linux-iommu/[email protected]/
The solution is to enable it under DMA API where IDXD shared workqueue users
can use ENQCMDS to submit work on buffers mapped by DMA API.
This patchset adds support for attaching PASID to the device's default
domain and the ability to allocate global PASIDs from IOMMU APIs. IDXD driver
can then re-enable the kernel work queues and use them under DMA API.
This depends on the IOASID removal series. (merged)
https://lore.kernel.org/all/[email protected]/
Thanks,
Jacob
---
Changelog:
v10:
- Fix global PASID alloc function with device's max_pasid=0
v9:
- Fix an IDXD driver issue where user interrupt enable bit got cleared
during device enable/disable cycle. Reported and tested by
Tony Zhu <[email protected]>
- Rebased to v6.4-rc7
v8:
- further vt-d driver refactoring (3-6) around set/remove device PASID
(Baolu)
- make consistent use of NO_PASID in SMMU code (Jean)
- fix off-by-one error in max PASID check (Kevin)
v7:
- renamed IOMMU_DEF_RID_PASID to be IOMMU_NO_PASID to be more generic
(Jean)
- simplify range checking for sva PASID (Baolu)
v6:
- use a simplified version of vt-d driver change for set_device_pasid
from Baolu.
- check and rename global PASID allocation base
v5:
- exclude two patches related to supervisor mode, taken by VT-d
maintainer Baolu.
- move PASID range check into allocation API so that device drivers
only need to pass in struct device*. (Kevin)
- factor out helper functions in device-domain attach (Baolu)
- make explicit use of RID_PASID across architectures
v4:
- move dummy functions outside ifdef CONFIG_IOMMU_SVA (Baolu)
- dropped domain type check while disabling idxd system PASID (Baolu)
v3:
- moved global PASID allocation API from SVA to IOMMU (Kevin)
- remove #ifdef around global PASID reservation during boot (Baolu)
- remove restriction on PASID 0 allocation (Baolu)
- fix a bug in sysfs domain change when attaching devices
- clear idxd user interrupt enable bit after disabling device( Fenghua)
v2:
- refactored device PASID attach domain ops based on Baolu's early patch
- addressed TLB flush gap
- explicitly reserve RID_PASID from SVA PASID number space
- get dma domain directly, avoid checking domain types
Jacob Pan (3):
iommu: Generalize PASID 0 for normal DMA w/o PASID
iommu: Move global PASID allocation from SVA to core
dmaengine/idxd: Re-enable kernel workqueue under DMA API
Lu Baolu (4):
iommu/vt-d: Add domain_flush_pasid_iotlb()
iommu/vt-d: Remove pasid_mutex
iommu/vt-d: Make prq draining code generic
iommu/vt-d: Add set_dev_pasid callback for dma domain
drivers/dma/idxd/device.c | 39 ++---
drivers/dma/idxd/dma.c | 5 +-
drivers/dma/idxd/idxd.h | 9 +
drivers/dma/idxd/init.c | 54 +++++-
drivers/dma/idxd/sysfs.c | 7 -
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 +-
drivers/iommu/intel/iommu.c | 161 +++++++++++++++---
drivers/iommu/intel/iommu.h | 9 +
drivers/iommu/intel/pasid.c | 2 +-
drivers/iommu/intel/pasid.h | 2 -
drivers/iommu/intel/svm.c | 53 +-----
drivers/iommu/iommu-sva.c | 28 ++-
drivers/iommu/iommu.c | 28 +++
include/linux/iommu.h | 11 ++
15 files changed, 291 insertions(+), 135 deletions(-)
--
2.25.1
Global PASID can be used beyond SVA. For example, drivers that use
Intel ENQCMD to submit work must use global PASIDs in that PASID
is stored in a per CPU MSR. When such device need to submit work
for in-kernel DMA with PASID, it must allocate PASIDs from the same
global number space to avoid conflict.
This patch moves global PASID allocation APIs from SVA to IOMMU APIs.
Reserved PASIDs, currently only RID_PASID, are excluded from the global
PASID allocation.
It is expected that device drivers will use the allocated PASIDs to
attach to appropriate IOMMU domains for use.
Reviewed-by: Lu Baolu <[email protected]>
Reviewed-by: Kevin Tian <[email protected]>
Signed-off-by: Jacob Pan <[email protected]>
---
v8: fix off-by-one in max_pasid check
v7: simplify range check (Baolu)
v6: explicitly exclude reserved a range from SVA PASID allocation
check mm PASID compatibility with device
v5: move PASID range check inside API so that device drivers only pass
in struct device* (Kevin)
v4: move dummy functions outside ifdef CONFIG_IOMMU_SVA (Baolu)
---
drivers/iommu/iommu-sva.c | 28 ++++++++++------------------
drivers/iommu/iommu.c | 28 ++++++++++++++++++++++++++++
include/linux/iommu.h | 10 ++++++++++
3 files changed, 48 insertions(+), 18 deletions(-)
diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
index 3ebd4b6586b3..8627acc6e60c 100644
--- a/drivers/iommu/iommu-sva.c
+++ b/drivers/iommu/iommu-sva.c
@@ -10,33 +10,30 @@
#include "iommu-sva.h"
static DEFINE_MUTEX(iommu_sva_lock);
-static DEFINE_IDA(iommu_global_pasid_ida);
/* Allocate a PASID for the mm within range (inclusive) */
-static int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max)
+static int iommu_sva_alloc_pasid(struct mm_struct *mm, struct device *dev)
{
+ ioasid_t pasid;
int ret = 0;
- if (min == IOMMU_PASID_INVALID ||
- max == IOMMU_PASID_INVALID ||
- min == 0 || max < min)
- return -EINVAL;
-
if (!arch_pgtable_dma_compat(mm))
return -EBUSY;
mutex_lock(&iommu_sva_lock);
/* Is a PASID already associated with this mm? */
if (mm_valid_pasid(mm)) {
- if (mm->pasid < min || mm->pasid > max)
+ if (mm->pasid >= dev->iommu->max_pasids)
ret = -EOVERFLOW;
goto out;
}
- ret = ida_alloc_range(&iommu_global_pasid_ida, min, max, GFP_KERNEL);
- if (ret < min)
+ pasid = iommu_alloc_global_pasid_dev(dev);
+ if (pasid == IOMMU_PASID_INVALID) {
+ ret = -ENOSPC;
goto out;
- mm->pasid = ret;
+ }
+ mm->pasid = pasid;
ret = 0;
out:
mutex_unlock(&iommu_sva_lock);
@@ -63,15 +60,10 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
{
struct iommu_domain *domain;
struct iommu_sva *handle;
- ioasid_t max_pasids;
int ret;
- max_pasids = dev->iommu->max_pasids;
- if (!max_pasids)
- return ERR_PTR(-EOPNOTSUPP);
-
/* Allocate mm->pasid if necessary. */
- ret = iommu_sva_alloc_pasid(mm, 1, max_pasids - 1);
+ ret = iommu_sva_alloc_pasid(mm, dev);
if (ret)
return ERR_PTR(ret);
@@ -216,5 +208,5 @@ void mm_pasid_drop(struct mm_struct *mm)
if (likely(!mm_valid_pasid(mm)))
return;
- ida_free(&iommu_global_pasid_ida, mm->pasid);
+ iommu_free_global_pasid(mm->pasid);
}
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index da340f11c5f5..37d5e1b8ed01 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -39,6 +39,7 @@
static struct kset *iommu_group_kset;
static DEFINE_IDA(iommu_group_ida);
+static DEFINE_IDA(iommu_global_pasid_ida);
static unsigned int iommu_def_domain_type __read_mostly;
static bool iommu_dma_strict __read_mostly = IS_ENABLED(CONFIG_IOMMU_DEFAULT_DMA_STRICT);
@@ -3399,3 +3400,30 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
return domain;
}
+
+ioasid_t iommu_alloc_global_pasid_dev(struct device *dev)
+{
+ int ret;
+
+ /* max_pasids == 0 means that the device does not support PASID */
+ if (!dev->iommu->max_pasids)
+ return IOMMU_PASID_INVALID;
+
+ /*
+ * max_pasids is set up by vendor driver based on number of PASID bits
+ * supported but the IDA allocation is inclusive.
+ */
+ ret = ida_alloc_range(&iommu_global_pasid_ida, IOMMU_FIRST_GLOBAL_PASID,
+ dev->iommu->max_pasids - 1, GFP_KERNEL);
+ return ret < 0 ? IOMMU_PASID_INVALID : ret;
+}
+EXPORT_SYMBOL_GPL(iommu_alloc_global_pasid_dev);
+
+void iommu_free_global_pasid(ioasid_t pasid)
+{
+ if (WARN_ON(pasid == IOMMU_PASID_INVALID))
+ return;
+
+ ida_free(&iommu_global_pasid_ida, pasid);
+}
+EXPORT_SYMBOL_GPL(iommu_free_global_pasid);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 2870bc29d456..e97148f31038 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -197,6 +197,7 @@ enum iommu_dev_features {
};
#define IOMMU_NO_PASID (0U) /* Reserved for DMA w/o PASID */
+#define IOMMU_FIRST_GLOBAL_PASID (1U) /*starting range for allocation */
#define IOMMU_PASID_INVALID (-1U)
typedef unsigned int ioasid_t;
@@ -728,6 +729,8 @@ void iommu_detach_device_pasid(struct iommu_domain *domain,
struct iommu_domain *
iommu_get_domain_for_dev_pasid(struct device *dev, ioasid_t pasid,
unsigned int type);
+ioasid_t iommu_alloc_global_pasid_dev(struct device *dev);
+void iommu_free_global_pasid(ioasid_t pasid);
#else /* CONFIG_IOMMU_API */
struct iommu_ops {};
@@ -1089,6 +1092,13 @@ iommu_get_domain_for_dev_pasid(struct device *dev, ioasid_t pasid,
{
return NULL;
}
+
+static inline ioasid_t iommu_alloc_global_pasid_dev(struct device *dev)
+{
+ return IOMMU_PASID_INVALID;
+}
+
+static inline void iommu_free_global_pasid(ioasid_t pasid) {}
#endif /* CONFIG_IOMMU_API */
/**
--
2.25.1
> From: Jacob Pan <[email protected]>
> Sent: Thursday, July 13, 2023 12:34 AM
>
> }
> +
> +ioasid_t iommu_alloc_global_pasid_dev(struct device *dev)
> +{
> + int ret;
> +
> + /* max_pasids == 0 means that the device does not support PASID */
> + if (!dev->iommu->max_pasids)
> + return IOMMU_PASID_INVALID;
> +
> + /*
> + * max_pasids is set up by vendor driver based on number of PASID
> bits
> + * supported but the IDA allocation is inclusive.
> + */
> + ret = ida_alloc_range(&iommu_global_pasid_ida,
> IOMMU_FIRST_GLOBAL_PASID,
> + dev->iommu->max_pasids - 1, GFP_KERNEL);
> + return ret < 0 ? IOMMU_PASID_INVALID : ret;
> +}
> +EXPORT_SYMBOL_GPL(iommu_alloc_global_pasid_dev);
nit. Just call it iommu_alloc_global_pasid.
Hi Kevin,
On Thu, 13 Jul 2023 07:42:01 +0000, "Tian, Kevin" <[email protected]>
wrote:
> > From: Jacob Pan <[email protected]>
> > Sent: Thursday, July 13, 2023 12:34 AM
> >
> > }
> > +
> > +ioasid_t iommu_alloc_global_pasid_dev(struct device *dev)
> > +{
> > + int ret;
> > +
> > + /* max_pasids == 0 means that the device does not support
> > PASID */
> > + if (!dev->iommu->max_pasids)
> > + return IOMMU_PASID_INVALID;
> > +
> > + /*
> > + * max_pasids is set up by vendor driver based on number of
> > PASID bits
> > + * supported but the IDA allocation is inclusive.
> > + */
> > + ret = ida_alloc_range(&iommu_global_pasid_ida,
> > IOMMU_FIRST_GLOBAL_PASID,
> > + dev->iommu->max_pasids - 1, GFP_KERNEL);
> > + return ret < 0 ? IOMMU_PASID_INVALID : ret;
> > +}
> > +EXPORT_SYMBOL_GPL(iommu_alloc_global_pasid_dev);
>
> nit. Just call it iommu_alloc_global_pasid.
will do. it will be symmetric.
Thanks,
Jacob