Hi Robin, Will and everyone,
I think this series is relative mature now, please give your valuable suggestions,
thanks!
This patch series is split from the series[1] that containes both IOMMU part and
VFIO part. The VFIO part will be sent out in another series.
[1] https://lore.kernel.org/linux-iommu/[email protected]/
changelog:
v4:
- Modify the framework as suggested by Baolu, thanks!
- Add trace for iommu ops.
- Extract io-pgtable part.
v3:
- Merge start_dirty_log and stop_dirty_log into switch_dirty_log. (Yi Sun)
- Maintain the dirty log status in iommu_domain.
- Update commit message to make patch easier to review.
v2:
- Address all comments of RFC version, thanks for all of you ;-)
- Add a bugfix that start dirty log for newly added dma ranges and domain.
Hi everyone,
This patch series introduces a framework of iommu dirty log tracking, and smmuv3
realizes this framework. This new feature can be used by VFIO dma dirty tracking.
Intention：
Some types of IOMMU are capable of tracking DMA dirty log, such as
ARM SMMU with HTTU or Intel IOMMU with SLADE. This introduces the
dirty log tracking framework in the IOMMU base layer.
Three new essential interfaces are added, and we maintaince the status
of dirty log tracking in iommu_domain.
1. iommu_switch_dirty_log: Perform actions to start|stop dirty log tracking
2. iommu_sync_dirty_log: Sync dirty log from IOMMU into a dirty bitmap
3. iommu_clear_dirty_log: Clear dirty log of IOMMU by a mask bitmap
About SMMU HTTU:
HTTU (Hardware Translation Table Update) is a feature of ARM SMMUv3, it can update
access flag or/and dirty state of the TTD (Translation Table Descriptor) by hardware.
With HTTU, stage1 TTD is classified into 3 types:
DBM bit AP[2](readonly bit)
1. writable_clean 1 1
2. writable_dirty 1 0
3. readonly 0 1
If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from writable_clean to
writable_dirty. Then software can scan TTD to sync dirty state into dirty bitmap. With
this feature, we can track the dirty log of DMA continuously and precisely.
About this series:
Patch 1-3：Introduce dirty log tracking framework in the IOMMU base layer, and two common
interfaces that can be used by many types of iommu.
Patch 4-6: Add feature detection for smmu HTTU and enable HTTU for smmu stage1 mapping.
And add feature detection for smmu BBML. We need to split block mapping when
start dirty log tracking and merge page mapping when stop dirty log tracking,
which requires break-before-make procedure. But it might cause problems when the
TTD is alive. The I/O streams might not tolerate translation faults. So BBML
should be used.
Patch 7-12: We implement these interfaces for arm smmuv3.
Thanks,
Keqian
Jean-Philippe Brucker (1):
iommu/arm-smmu-v3: Add support for Hardware Translation Table Update
Keqian Zhu (1):
iommu: Introduce dirty log tracking framework
Kunkun Jiang (11):
iommu/io-pgtable-arm: Add quirk ARM_HD and ARM_BBMLx
iommu/io-pgtable-arm: Add and realize split_block ops
iommu/io-pgtable-arm: Add and realize merge_page ops
iommu/io-pgtable-arm: Add and realize sync_dirty_log ops
iommu/io-pgtable-arm: Add and realize clear_dirty_log ops
iommu/arm-smmu-v3: Enable HTTU for stage1 with io-pgtable mapping
iommu/arm-smmu-v3: Add feature detection for BBML
iommu/arm-smmu-v3: Realize switch_dirty_log iommu ops
iommu/arm-smmu-v3: Realize sync_dirty_log iommu ops
iommu/arm-smmu-v3: Realize clear_dirty_log iommu ops
iommu/arm-smmu-v3: Realize support_dirty_log iommu ops
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 268 +++++++++++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +
drivers/iommu/io-pgtable-arm.c | 389 +++++++++++++++++-
drivers/iommu/iommu.c | 206 +++++++++-
include/linux/io-pgtable.h | 23 ++
include/linux/iommu.h | 65 +++
include/trace/events/iommu.h | 63 +++
8 files changed, 1026 insertions(+), 4 deletions(-)
--
2.19.1
From: Kunkun Jiang <[email protected]>
This detects BBML feature and if SMMU supports it, transfer BBMLx
quirk to io-pgtable.
Co-developed-by: Keqian Zhu <[email protected]>
Signed-off-by: Kunkun Jiang <[email protected]>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 19 +++++++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 ++++++
2 files changed, 25 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index c42e59655fd0..3a2dc3177180 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2051,6 +2051,11 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,
if (smmu->features & ARM_SMMU_FEAT_HD)
pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_ARM_HD;
+ if (smmu->features & ARM_SMMU_FEAT_BBML1)
+ pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_ARM_BBML1;
+ else if (smmu->features & ARM_SMMU_FEAT_BBML2)
+ pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_ARM_BBML2;
+
pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain);
if (!pgtbl_ops)
return -ENOMEM;
@@ -3419,6 +3424,20 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
/* IDR3 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR3);
+ switch (FIELD_GET(IDR3_BBML, reg)) {
+ case IDR3_BBML0:
+ break;
+ case IDR3_BBML1:
+ smmu->features |= ARM_SMMU_FEAT_BBML1;
+ break;
+ case IDR3_BBML2:
+ smmu->features |= ARM_SMMU_FEAT_BBML2;
+ break;
+ default:
+ dev_err(smmu->dev, "unknown/unsupported BBM behavior level\n");
+ return -ENXIO;
+ }
+
if (FIELD_GET(IDR3_RIL, reg))
smmu->features |= ARM_SMMU_FEAT_RANGE_INV;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 3edcd31b046e..e3b6bdd292c9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -54,6 +54,10 @@
#define IDR1_SIDSIZE GENMASK(5, 0)
#define ARM_SMMU_IDR3 0xc
+#define IDR3_BBML GENMASK(12, 11)
+#define IDR3_BBML0 0
+#define IDR3_BBML1 1
+#define IDR3_BBML2 2
#define IDR3_RIL (1 << 10)
#define ARM_SMMU_IDR5 0x14
@@ -613,6 +617,8 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_E2H (1 << 18)
#define ARM_SMMU_FEAT_HA (1 << 19)
#define ARM_SMMU_FEAT_HD (1 << 20)
+#define ARM_SMMU_FEAT_BBML1 (1 << 21)
+#define ARM_SMMU_FEAT_BBML2 (1 << 22)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
--
2.19.1
From: Kunkun Jiang <[email protected]>
As nested mode is not upstreamed now, we just aim to support dirty
log tracking for stage1 with io-pgtable mapping (means not support
SVA mapping). If HTTU is supported, we enable HA/HD bits in the SMMU
CD and transfer ARM_HD quirk to io-pgtable.
Co-developed-by: Keqian Zhu <[email protected]>
Signed-off-by: Kunkun Jiang <[email protected]>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 4ac59a89bc76..c42e59655fd0 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1942,6 +1942,7 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain,
FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
+ CTXDESC_CD_0_TCR_HA | CTXDESC_CD_0_TCR_HD |
CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
cfg->cd.mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
@@ -2047,6 +2048,8 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,
if (!iommu_get_dma_strict(domain))
pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_NON_STRICT;
+ if (smmu->features & ARM_SMMU_FEAT_HD)
+ pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_ARM_HD;
pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain);
if (!pgtbl_ops)
--
2.19.1
From: Kunkun Jiang <[email protected]>
We have implemented these interfaces required to support iommu
dirty log tracking. The last step is reporting this feature to
upper user, then the user can perform higher policy base on it.
For arm smmuv3, it is equal to ARM_SMMU_FEAT_HD.
Co-developed-by: Keqian Zhu <[email protected]>
Signed-off-by: Kunkun Jiang <[email protected]>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 9b4739247dbb..59d11f084199 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2684,6 +2684,13 @@ static int arm_smmu_merge_page(struct iommu_domain *domain, unsigned long iova,
return ret;
}
+static bool arm_smmu_support_dirty_log(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ return !!(smmu_domain->smmu->features & ARM_SMMU_FEAT_HD);
+}
+
static int arm_smmu_switch_dirty_log(struct iommu_domain *domain, bool enable,
unsigned long iova, size_t size, int prot)
{
@@ -2872,6 +2879,7 @@ static struct iommu_ops arm_smmu_ops = {
.release_device = arm_smmu_release_device,
.device_group = arm_smmu_device_group,
.enable_nesting = arm_smmu_enable_nesting,
+ .support_dirty_log = arm_smmu_support_dirty_log,
.switch_dirty_log = arm_smmu_switch_dirty_log,
.sync_dirty_log = arm_smmu_sync_dirty_log,
.clear_dirty_log = arm_smmu_clear_dirty_log,
--
2.19.1
From: Jean-Philippe Brucker <[email protected]>
If the SMMU supports it and the kernel was built with HTTU support,
enable hardware update of access and dirty flags. This is essential for
shared page tables, to reduce the number of access faults on the fault
queue. Normal DMA with io-pgtables doesn't currently use the access or
dirty flags.
We can enable HTTU even if CPUs don't support it, because the kernel
always checks for HW dirty bit and updates the PTE flags atomically.
Signed-off-by: Jean-Philippe Brucker <[email protected]>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 41 ++++++++++++++++++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 ++++
3 files changed, 50 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index bb251cab61f3..ae075e675892 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -121,10 +121,12 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
if (err)
goto out_free_asid;
+ /* HA and HD will be filtered out later if not supported by the SMMU */
tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) |
FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) |
FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) |
FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) |
+ CTXDESC_CD_0_TCR_HA | CTXDESC_CD_0_TCR_HD |
CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
switch (PAGE_SIZE) {
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 54b2f27b81d4..4ac59a89bc76 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1010,10 +1010,17 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid,
* this substream's traffic
*/
} else { /* (1) and (2) */
+ u64 tcr = cd->tcr;
+
cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
cdptr[2] = 0;
cdptr[3] = cpu_to_le64(cd->mair);
+ if (!(smmu->features & ARM_SMMU_FEAT_HD))
+ tcr &= ~CTXDESC_CD_0_TCR_HD;
+ if (!(smmu->features & ARM_SMMU_FEAT_HA))
+ tcr &= ~CTXDESC_CD_0_TCR_HA;
+
/*
* STE is live, and the SMMU might read dwords of this CD in any
* order. Ensure that it observes valid values before reading
@@ -1021,7 +1028,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid,
*/
arm_smmu_sync_cd(smmu_domain, ssid, true);
- val = cd->tcr |
+ val = tcr |
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
#endif
@@ -3242,6 +3249,28 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
return 0;
}
+static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg)
+{
+ u32 fw_features = smmu->features & (ARM_SMMU_FEAT_HA | ARM_SMMU_FEAT_HD);
+ u32 features = 0;
+
+ switch (FIELD_GET(IDR0_HTTU, reg)) {
+ case IDR0_HTTU_ACCESS_DIRTY:
+ features |= ARM_SMMU_FEAT_HD;
+ fallthrough;
+ case IDR0_HTTU_ACCESS:
+ features |= ARM_SMMU_FEAT_HA;
+ }
+
+ if (smmu->dev->of_node)
+ smmu->features |= features;
+ else if (features != fw_features)
+ /* ACPI IORT sets the HTTU bits */
+ dev_warn(smmu->dev,
+ "IDR0.HTTU overridden by FW configuration (0x%x)\n",
+ fw_features);
+}
+
static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
{
u32 reg;
@@ -3302,6 +3331,8 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
smmu->features |= ARM_SMMU_FEAT_E2H;
}
+ arm_smmu_get_httu(smmu, reg);
+
/*
* The coherency feature as set by FW is used in preference to the ID
* register, but warn on mismatch.
@@ -3487,6 +3518,14 @@ static int arm_smmu_device_acpi_probe(struct platform_device *pdev,
if (iort_smmu->flags & ACPI_IORT_SMMU_V3_COHACC_OVERRIDE)
smmu->features |= ARM_SMMU_FEAT_COHERENCY;
+ switch (FIELD_GET(ACPI_IORT_SMMU_V3_HTTU_OVERRIDE, iort_smmu->flags)) {
+ case IDR0_HTTU_ACCESS_DIRTY:
+ smmu->features |= ARM_SMMU_FEAT_HD;
+ fallthrough;
+ case IDR0_HTTU_ACCESS:
+ smmu->features |= ARM_SMMU_FEAT_HA;
+ }
+
return 0;
}
#else
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 46e8c49214a8..3edcd31b046e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -33,6 +33,9 @@
#define IDR0_ASID16 (1 << 12)
#define IDR0_ATS (1 << 10)
#define IDR0_HYP (1 << 9)
+#define IDR0_HTTU GENMASK(7, 6)
+#define IDR0_HTTU_ACCESS 1
+#define IDR0_HTTU_ACCESS_DIRTY 2
#define IDR0_COHACC (1 << 4)
#define IDR0_TTF GENMASK(3, 2)
#define IDR0_TTF_AARCH64 2
@@ -285,6 +288,9 @@
#define CTXDESC_CD_0_TCR_IPS GENMASK_ULL(34, 32)
#define CTXDESC_CD_0_TCR_TBI0 (1ULL << 38)
+#define CTXDESC_CD_0_TCR_HA (1UL << 43)
+#define CTXDESC_CD_0_TCR_HD (1UL << 42)
+
#define CTXDESC_CD_0_AA64 (1UL << 41)
#define CTXDESC_CD_0_S (1UL << 44)
#define CTXDESC_CD_0_R (1UL << 45)
@@ -605,6 +611,8 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_BTM (1 << 16)
#define ARM_SMMU_FEAT_SVA (1 << 17)
#define ARM_SMMU_FEAT_E2H (1 << 18)
+#define ARM_SMMU_FEAT_HA (1 << 19)
+#define ARM_SMMU_FEAT_HD (1 << 20)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
--
2.19.1
From: Kunkun Jiang <[email protected]>
This realizes switch_dirty_log. In order to get finer dirty
granule, it invokes arm_smmu_split_block when start dirty
log, and invokes arm_smmu_merge_page() to recover block
mapping when stop dirty log.
Co-developed-by: Keqian Zhu <[email protected]>
Signed-off-by: Kunkun Jiang <[email protected]>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 142 ++++++++++++++++++++
drivers/iommu/iommu.c | 5 +-
include/linux/iommu.h | 2 +
3 files changed, 147 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 3a2dc3177180..6de81d6ab652 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2580,6 +2580,147 @@ static int arm_smmu_enable_nesting(struct iommu_domain *domain)
return ret;
}
+static int arm_smmu_split_block(struct iommu_domain *domain,
+ unsigned long iova, size_t size)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
+ size_t handled_size;
+
+ if (!(smmu->features & (ARM_SMMU_FEAT_BBML1 | ARM_SMMU_FEAT_BBML2))) {
+ dev_err(smmu->dev, "don't support BBML1/2, can't split block\n");
+ return -ENODEV;
+ }
+ if (!ops || !ops->split_block) {
+ pr_err("io-pgtable don't realize split block\n");
+ return -ENODEV;
+ }
+
+ handled_size = ops->split_block(ops, iova, size);
+ if (handled_size != size) {
+ pr_err("split block failed\n");
+ return -EFAULT;
+ }
+
+ return 0;
+}
+
+static int __arm_smmu_merge_page(struct iommu_domain *domain,
+ unsigned long iova, phys_addr_t paddr,
+ size_t size, int prot)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
+ size_t handled_size;
+
+ if (!ops || !ops->merge_page) {
+ pr_err("io-pgtable don't realize merge page\n");
+ return -ENODEV;
+ }
+
+ while (size) {
+ size_t pgsize = iommu_pgsize(domain, iova | paddr, size);
+
+ handled_size = ops->merge_page(ops, iova, paddr, pgsize, prot);
+ if (handled_size != pgsize) {
+ pr_err("merge page failed\n");
+ return -EFAULT;
+ }
+
+ pr_debug("merge handled: iova 0x%lx pa %pa size 0x%zx\n",
+ iova, &paddr, pgsize);
+
+ iova += pgsize;
+ paddr += pgsize;
+ size -= pgsize;
+ }
+
+ return 0;
+}
+
+static int arm_smmu_merge_page(struct iommu_domain *domain, unsigned long iova,
+ size_t size, int prot)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
+ phys_addr_t phys;
+ dma_addr_t p, i;
+ size_t cont_size;
+ int ret = 0;
+
+ if (!(smmu->features & (ARM_SMMU_FEAT_BBML1 | ARM_SMMU_FEAT_BBML2))) {
+ dev_err(smmu->dev, "don't support BBML1/2, can't merge page\n");
+ return -ENODEV;
+ }
+
+ if (!ops || !ops->iova_to_phys)
+ return -ENODEV;
+
+ while (size) {
+ phys = ops->iova_to_phys(ops, iova);
+ cont_size = PAGE_SIZE;
+ p = phys + cont_size;
+ i = iova + cont_size;
+
+ while (cont_size < size && p == ops->iova_to_phys(ops, i)) {
+ p += PAGE_SIZE;
+ i += PAGE_SIZE;
+ cont_size += PAGE_SIZE;
+ }
+
+ if (cont_size != PAGE_SIZE) {
+ ret = __arm_smmu_merge_page(domain, iova, phys,
+ cont_size, prot);
+ if (ret)
+ break;
+ }
+
+ iova += cont_size;
+ size -= cont_size;
+ }
+
+ return ret;
+}
+
+static int arm_smmu_switch_dirty_log(struct iommu_domain *domain, bool enable,
+ unsigned long iova, size_t size, int prot)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+ if (!(smmu->features & ARM_SMMU_FEAT_HD))
+ return -ENODEV;
+ if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
+ return -EINVAL;
+
+ if (enable) {
+ /*
+ * For SMMU, the hardware dirty management is always enabled if
+ * hardware supports HTTU HD. The action to start dirty log is
+ * spliting block mapping.
+ *
+ * We don't return error even if the split operation fail, as we
+ * can still track dirty at block granule, which is still a much
+ * better choice compared to full dirty policy.
+ */
+ arm_smmu_split_block(domain, iova, size);
+ } else {
+ /*
+ * For SMMU, the hardware dirty management is always enabled if
+ * hardware supports HTTU HD. The action to stop dirty log is
+ * merging page mapping.
+ *
+ * We don't return error even if the merge operation fail, as it
+ * just effects performace of DMA transaction.
+ */
+ arm_smmu_merge_page(domain, iova, size, prot);
+ }
+
+ return 0;
+}
+
static int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
{
return iommu_fwspec_add_ids(dev, args->args, 1);
@@ -2678,6 +2819,7 @@ static struct iommu_ops arm_smmu_ops = {
.release_device = arm_smmu_release_device,
.device_group = arm_smmu_device_group,
.enable_nesting = arm_smmu_enable_nesting,
+ .switch_dirty_log = arm_smmu_switch_dirty_log,
.of_xlate = arm_smmu_of_xlate,
.get_resv_regions = arm_smmu_get_resv_regions,
.put_resv_regions = generic_iommu_put_resv_regions,
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 0d15620d1e90..bb19df2317ed 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2375,8 +2375,8 @@ phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova)
}
EXPORT_SYMBOL_GPL(iommu_iova_to_phys);
-static size_t iommu_pgsize(struct iommu_domain *domain,
- unsigned long addr_merge, size_t size)
+size_t iommu_pgsize(struct iommu_domain *domain,
+ unsigned long addr_merge, size_t size)
{
unsigned int pgsize_idx;
size_t pgsize;
@@ -2406,6 +2406,7 @@ static size_t iommu_pgsize(struct iommu_domain *domain,
return pgsize;
}
+EXPORT_SYMBOL_GPL(iommu_pgsize);
static int __iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index e0e40dda974d..0a77db4f397f 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -427,6 +427,8 @@ extern int iommu_sva_unbind_gpasid(struct iommu_domain *domain,
struct device *dev, ioasid_t pasid);
extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
+extern size_t iommu_pgsize(struct iommu_domain *domain,
+ unsigned long addr_merge, size_t size);
extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot);
extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova,
--
2.19.1
From: Kunkun Jiang <[email protected]>
This realizes sync_dirty_log iommu ops based on sync_dirty_log
io-pgtable ops.
Co-developed-by: Keqian Zhu <[email protected]>
Signed-off-by: Kunkun Jiang <[email protected]>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 30 +++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6de81d6ab652..3d3c0f8e2446 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2721,6 +2721,35 @@ static int arm_smmu_switch_dirty_log(struct iommu_domain *domain, bool enable,
return 0;
}
+static int arm_smmu_sync_dirty_log(struct iommu_domain *domain,
+ unsigned long iova, size_t size,
+ unsigned long *bitmap,
+ unsigned long base_iova,
+ unsigned long bitmap_pgshift)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+ if (!(smmu->features & ARM_SMMU_FEAT_HD))
+ return -ENODEV;
+ if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
+ return -EINVAL;
+
+ if (!ops || !ops->sync_dirty_log) {
+ pr_err("io-pgtable don't realize sync dirty log\n");
+ return -ENODEV;
+ }
+
+ /*
+ * Flush iotlb to ensure all inflight transactions are completed.
+ * See doc IHI0070Da 3.13.4 "HTTU behavior summary".
+ */
+ arm_smmu_flush_iotlb_all(domain);
+ return ops->sync_dirty_log(ops, iova, size, bitmap, base_iova,
+ bitmap_pgshift);
+}
+
static int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
{
return iommu_fwspec_add_ids(dev, args->args, 1);
@@ -2820,6 +2849,7 @@ static struct iommu_ops arm_smmu_ops = {
.device_group = arm_smmu_device_group,
.enable_nesting = arm_smmu_enable_nesting,
.switch_dirty_log = arm_smmu_switch_dirty_log,
+ .sync_dirty_log = arm_smmu_sync_dirty_log,
.of_xlate = arm_smmu_of_xlate,
.get_resv_regions = arm_smmu_get_resv_regions,
.put_resv_regions = generic_iommu_put_resv_regions,
--
2.19.1
Hi all,
The VFIO part is at here: https://lore.kernel.org/kvm/[email protected]/
Thanks,
Keqian
On 2021/5/7 18:21, Keqian Zhu wrote:
> Hi Robin, Will and everyone,
>
> I think this series is relative mature now, please give your valuable suggestions,
> thanks!
>
>
> This patch series is split from the series[1] that containes both IOMMU part and
> VFIO part. The VFIO part will be sent out in another series.
>
> [1] https://lore.kernel.org/linux-iommu/[email protected]/
>
> changelog:
>
> v4:
> - Modify the framework as suggested by Baolu, thanks!
> - Add trace for iommu ops.
> - Extract io-pgtable part.
>
> v3:
> - Merge start_dirty_log and stop_dirty_log into switch_dirty_log. (Yi Sun)
> - Maintain the dirty log status in iommu_domain.
> - Update commit message to make patch easier to review.
>
> v2:
> - Address all comments of RFC version, thanks for all of you ;-)
> - Add a bugfix that start dirty log for newly added dma ranges and domain.
>
>
>
> Hi everyone,
>
> This patch series introduces a framework of iommu dirty log tracking, and smmuv3
> realizes this framework. This new feature can be used by VFIO dma dirty tracking.
>
> Intention：
>
> Some types of IOMMU are capable of tracking DMA dirty log, such as
> ARM SMMU with HTTU or Intel IOMMU with SLADE. This introduces the
> dirty log tracking framework in the IOMMU base layer.
>
> Three new essential interfaces are added, and we maintaince the status
> of dirty log tracking in iommu_domain.
> 1. iommu_switch_dirty_log: Perform actions to start|stop dirty log tracking
> 2. iommu_sync_dirty_log: Sync dirty log from IOMMU into a dirty bitmap
> 3. iommu_clear_dirty_log: Clear dirty log of IOMMU by a mask bitmap
>
> About SMMU HTTU:
>
> HTTU (Hardware Translation Table Update) is a feature of ARM SMMUv3, it can update
> access flag or/and dirty state of the TTD (Translation Table Descriptor) by hardware.
> With HTTU, stage1 TTD is classified into 3 types:
> DBM bit AP[2](readonly bit)
> 1. writable_clean 1 1
> 2. writable_dirty 1 0
> 3. readonly 0 1
>
> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from writable_clean to
> writable_dirty. Then software can scan TTD to sync dirty state into dirty bitmap. With
> this feature, we can track the dirty log of DMA continuously and precisely.
>
> About this series:
>
> Patch 1-3：Introduce dirty log tracking framework in the IOMMU base layer, and two common
> interfaces that can be used by many types of iommu.
>
> Patch 4-6: Add feature detection for smmu HTTU and enable HTTU for smmu stage1 mapping.
> And add feature detection for smmu BBML. We need to split block mapping when
> start dirty log tracking and merge page mapping when stop dirty log tracking,
> which requires break-before-make procedure. But it might cause problems when the
> TTD is alive. The I/O streams might not tolerate translation faults. So BBML
> should be used.
>
> Patch 7-12: We implement these interfaces for arm smmuv3.
>
> Thanks,
> Keqian
>
> Jean-Philippe Brucker (1):
> iommu/arm-smmu-v3: Add support for Hardware Translation Table Update
>
> Keqian Zhu (1):
> iommu: Introduce dirty log tracking framework
>
> Kunkun Jiang (11):
> iommu/io-pgtable-arm: Add quirk ARM_HD and ARM_BBMLx
> iommu/io-pgtable-arm: Add and realize split_block ops
> iommu/io-pgtable-arm: Add and realize merge_page ops
> iommu/io-pgtable-arm: Add and realize sync_dirty_log ops
> iommu/io-pgtable-arm: Add and realize clear_dirty_log ops
> iommu/arm-smmu-v3: Enable HTTU for stage1 with io-pgtable mapping
> iommu/arm-smmu-v3: Add feature detection for BBML
> iommu/arm-smmu-v3: Realize switch_dirty_log iommu ops
> iommu/arm-smmu-v3: Realize sync_dirty_log iommu ops
> iommu/arm-smmu-v3: Realize clear_dirty_log iommu ops
> iommu/arm-smmu-v3: Realize support_dirty_log iommu ops
>
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 268 +++++++++++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +
> drivers/iommu/io-pgtable-arm.c | 389 +++++++++++++++++-
> drivers/iommu/iommu.c | 206 +++++++++-
> include/linux/io-pgtable.h | 23 ++
> include/linux/iommu.h | 65 +++
> include/trace/events/iommu.h | 63 +++
> 8 files changed, 1026 insertions(+), 4 deletions(-)
>