LinuxLists.cc - [PATCH v9 0/5] iommu/arm-smmu: Add runtime pm/sleep support

[permalink] [raw]

Subject: [PATCH v9 4/5] iommu/arm-smmu: Add the device_link between masters and smmu

From: Sricharan R <[email protected]>

Finally add the device link between the master device and
smmu, so that the smmu gets runtime enabled/disabled only when the
master needs it. This is done from add_device callback which gets
called once when the master is added to the smmu.

Signed-off-by: Sricharan R <[email protected]>
Signed-off-by: Vivek Gautam <[email protected]>
Reviewed-by: Tomasz Figa <[email protected]>
---
drivers/iommu/arm-smmu.c | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 56a04ae80bf3..64953ff2281f 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1460,10 +1460,31 @@ static int arm_smmu_add_device(struct device *dev)

iommu_device_link(&smmu->iommu, dev);

+ if (pm_runtime_enabled(smmu->dev)) {
+ struct device_link *link;
+
+ /*
+ * Establish the link between smmu and master, so that the
+ * smmu gets runtime enabled/disabled as per the master's
+ * needs.
+ */
+ link = device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME);
+ if (!link) {
+ dev_warn(smmu->dev,
+ "Unable to add link to the consumer %s\n",
+ dev_name(dev));
+ ret = -ENODEV;
+ goto out_unlink;
+ }
+ }
+
arm_smmu_rpm_put(smmu);

return 0;

+out_unlink:
+ iommu_device_unlink(&smmu->iommu, dev);
+ arm_smmu_master_free_smes(fwspec);
out_rpm_put:
arm_smmu_rpm_put(smmu);
out_cfg_free:
@@ -1486,6 +1507,14 @@ static void arm_smmu_remove_device(struct device *dev)
cfg = fwspec->iommu_priv;
smmu = cfg->smmu;

+ if (pm_runtime_enabled(smmu->dev)) {
+ struct device_link *link;
+
+ link = device_link_find(dev, smmu->dev);
+ if (link)
+ device_link_del(link);
+ }
+
ret = arm_smmu_rpm_get(smmu);
if (ret < 0)
return;
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 08:58:18

[permalink] [raw]

Subject: [PATCH v9 5/5] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

qcom,smmu-v2 is an arm,smmu-v2 implementation with specific
clock and power requirements. This smmu core is used with
multiple masters on msm8996, viz. mdss, video, etc.
Add bindings for the same.

Signed-off-by: Vivek Gautam <[email protected]>
Reviewed-by: Rob Herring <[email protected]>
Reviewed-by: Tomasz Figa <[email protected]>
---
.../devicetree/bindings/iommu/arm,smmu.txt | 42 ++++++++++++++++++++++
drivers/iommu/arm-smmu.c | 14 ++++++++
2 files changed, 56 insertions(+)

diff --git a/Documentation/devicetree/bindings/iommu/arm,smmu.txt b/Documentation/devicetree/bindings/iommu/arm,smmu.txt
index 8a6ffce12af5..7c71a6ed465a 100644
--- a/Documentation/devicetree/bindings/iommu/arm,smmu.txt
+++ b/Documentation/devicetree/bindings/iommu/arm,smmu.txt
@@ -17,10 +17,19 @@ conditions.
"arm,mmu-401"
"arm,mmu-500"
"cavium,smmu-v2"
+ "qcom,<soc>-smmu-v2", "qcom,smmu-v2"

depending on the particular implementation and/or the
version of the architecture implemented.

+ A number of Qcom SoCs use qcom,smmu-v2 version of the IP.
+ "qcom,<soc>-smmu-v2" represents a soc specific compatible
+ string that should be present along with the "qcom,smmu-v2"
+ to facilitate SoC specific clocks/power connections and to
+ address specific bug fixes.
+ An example string would be -
+ "qcom,msm8996-smmu-v2", "qcom,smmu-v2".
+
- reg : Base address and size of the SMMU.

- #global-interrupts : The number of global interrupts exposed by the
@@ -71,6 +80,22 @@ conditions.
or using stream matching with #iommu-cells = <2>, and
may be ignored if present in such cases.

+- clock-names: List of the names of clocks input to the device. The
+ required list depends on particular implementation and
+ is as follows:
+ - for "qcom,smmu-v2":
+ - "bus": clock required for downstream bus access and
+ for the smmu ptw,
+ - "iface": clock required to access smmu's registers
+ through the TCU's programming interface.
+ - unspecified for other implementations.
+
+- clocks: Specifiers for all clocks listed in the clock-names property,
+ as per generic clock bindings.
+
+- power-domains: Specifiers for power domains required to be powered on for
+ the SMMU to operate, as per generic power domain bindings.
+
** Deprecated properties:

- mmu-masters (deprecated in favour of the generic "iommus" binding) :
@@ -137,3 +162,20 @@ conditions.
iommu-map = <0 &smmu3 0 0x400>;
...
};
+
+ /* Qcom's arm,smmu-v2 implementation */
+ smmu4: iommu {
+ compatible = "qcom,msm8996-smmu-v2", "qcom,smmu-v2";
+ reg = <0xd00000 0x10000>;
+
+ #global-interrupts = <1>;
+ interrupts = <GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 320 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 321 IRQ_TYPE_LEVEL_HIGH>;
+ #iommu-cells = <1>;
+ power-domains = <&mmcc MDSS_GDSC>;
+
+ clocks = <&mmcc SMMU_MDP_AXI_CLK>,
+ <&mmcc SMMU_MDP_AHB_CLK>;
+ clock-names = "bus", "iface";
+ };
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 64953ff2281f..1ef6ac56a347 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -119,6 +119,7 @@ enum arm_smmu_implementation {
GENERIC_SMMU,
ARM_MMU500,
CAVIUM_SMMUV2,
+ QCOM_SMMUV2,
};

struct arm_smmu_s2cr {
@@ -2000,6 +2001,17 @@ ARM_SMMU_MATCH_DATA(arm_mmu401, ARM_SMMU_V1_64K, GENERIC_SMMU);
ARM_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500);
ARM_SMMU_MATCH_DATA(cavium_smmuv2, ARM_SMMU_V2, CAVIUM_SMMUV2);

+static const char * const qcom_smmuv2_clks[] = {
+ "bus", "iface",
+};
+
+static const struct arm_smmu_match_data qcom_smmuv2 = {
+ .version = ARM_SMMU_V2,
+ .model = QCOM_SMMUV2,
+ .clks = qcom_smmuv2_clks,
+ .num_clks = ARRAY_SIZE(qcom_smmuv2_clks),
+};
+
static const struct of_device_id arm_smmu_of_match[] = {
{ .compatible = "arm,smmu-v1", .data = &smmu_generic_v1 },
{ .compatible = "arm,smmu-v2", .data = &smmu_generic_v2 },
@@ -2007,6 +2019,7 @@ static const struct of_device_id arm_smmu_of_match[] = {
{ .compatible = "arm,mmu-401", .data = &arm_mmu401 },
{ .compatible = "arm,mmu-500", .data = &arm_mmu500 },
{ .compatible = "cavium,smmu-v2", .data = &cavium_smmuv2 },
+ { .compatible = "qcom,smmu-v2", .data = &qcom_smmuv2 },
{ },
};
MODULE_DEVICE_TABLE(of, arm_smmu_of_match);
@@ -2381,6 +2394,7 @@ IOMMU_OF_DECLARE(arm_mmu400, "arm,mmu-400");
IOMMU_OF_DECLARE(arm_mmu401, "arm,mmu-401");
IOMMU_OF_DECLARE(arm_mmu500, "arm,mmu-500");
IOMMU_OF_DECLARE(cavium_smmuv2, "cavium,smmu-v2");
+IOMMU_OF_DECLARE(qcom_smmuv2, "qcom,smmu-v2");

MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
MODULE_AUTHOR("Will Deacon <[email protected]>");
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 08:58:40

[permalink] [raw]

Subject: [PATCH v9 2/5] iommu/arm-smmu: Add pm_runtime/sleep ops

From: Sricharan R <[email protected]>

The smmu needs to be functional only when the respective
master's using it are active. The device_link feature
helps to track such functional dependencies, so that the
iommu gets powered when the master device enables itself
using pm_runtime. So by adapting the smmu driver for
runtime pm, above said dependency can be addressed.

This patch adds the pm runtime/sleep callbacks to the
driver and also the functions to parse the smmu clocks
from DT and enable them in resume/suspend.

Signed-off-by: Sricharan R <[email protected]>
Signed-off-by: Archit Taneja <[email protected]>
[vivek: Clock rework to request bulk of clocks]
Signed-off-by: Vivek Gautam <[email protected]>
Reviewed-by: Tomasz Figa <[email protected]>
---
drivers/iommu/arm-smmu.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 58 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 69e7c60792a8..d5873d545024 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -48,6 +48,7 @@
#include <linux/of_iommu.h>
#include <linux/pci.h>
#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
#include <linux/slab.h>
#include <linux/spinlock.h>

@@ -205,6 +206,8 @@ struct arm_smmu_device {
u32 num_global_irqs;
u32 num_context_irqs;
unsigned int *irqs;
+ struct clk_bulk_data *clks;
+ int num_clks;

u32 cavium_id_base; /* Specific to Cavium */

@@ -1897,10 +1900,12 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
struct arm_smmu_match_data {
enum arm_smmu_arch_version version;
enum arm_smmu_implementation model;
+ const char * const *clks;
+ int num_clks;
};

#define ARM_SMMU_MATCH_DATA(name, ver, imp) \
-static struct arm_smmu_match_data name = { .version = ver, .model = imp }
+static const struct arm_smmu_match_data name = { .version = ver, .model = imp }

ARM_SMMU_MATCH_DATA(smmu_generic_v1, ARM_SMMU_V1, GENERIC_SMMU);
ARM_SMMU_MATCH_DATA(smmu_generic_v2, ARM_SMMU_V2, GENERIC_SMMU);
@@ -1919,6 +1924,23 @@ static const struct of_device_id arm_smmu_of_match[] = {
};
MODULE_DEVICE_TABLE(of, arm_smmu_of_match);

+static void arm_smmu_fill_clk_data(struct arm_smmu_device *smmu,
+ const char * const *clks)
+{
+ int i;
+
+ if (smmu->num_clks < 1)
+ return;
+
+ smmu->clks = devm_kcalloc(smmu->dev, smmu->num_clks,
+ sizeof(*smmu->clks), GFP_KERNEL);
+ if (!smmu->clks)
+ return;
+
+ for (i = 0; i < smmu->num_clks; i++)
+ smmu->clks[i].id = clks[i];
+}
+
#ifdef CONFIG_ACPI
static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
{
@@ -2001,6 +2023,9 @@ static int arm_smmu_device_dt_probe(struct platform_device *pdev,
data = of_device_get_match_data(dev);
smmu->version = data->version;
smmu->model = data->model;
+ smmu->num_clks = data->num_clks;
+
+ arm_smmu_fill_clk_data(smmu, data->clks);

parse_driver_options(smmu);

@@ -2099,6 +2124,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
smmu->irqs[i] = irq;
}

+ err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
+ if (err)
+ return err;
+
+ err = clk_bulk_prepare(smmu->num_clks, smmu->clks);
+ if (err)
+ return err;
+
err = arm_smmu_device_cfg_probe(smmu);
if (err)
return err;
@@ -2181,6 +2214,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)

/* Turn the thing off */
writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+
+ clk_bulk_unprepare(smmu->num_clks, smmu->clks);
+
return 0;
}

@@ -2197,7 +2233,27 @@ static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
return 0;
}

-static SIMPLE_DEV_PM_OPS(arm_smmu_pm_ops, NULL, arm_smmu_pm_resume);
+static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
+{
+ struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+
+ return clk_bulk_enable(smmu->num_clks, smmu->clks);
+}
+
+static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
+{
+ struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+
+ clk_bulk_disable(smmu->num_clks, smmu->clks);
+
+ return 0;
+}
+
+static const struct dev_pm_ops arm_smmu_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(NULL, arm_smmu_pm_resume)
+ SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend,
+ arm_smmu_runtime_resume, NULL)
+};

static struct platform_driver arm_smmu_driver = {
.driver = {
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 08:58:44

[permalink] [raw]

Subject: [PATCH v9 3/5] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

From: Sricharan R <[email protected]>

The smmu device probe/remove and add/remove master device callbacks
gets called when the smmu is not linked to its master, that is without
the context of the master device. So calling runtime apis in those places
separately.

Signed-off-by: Sricharan R <[email protected]>
[vivek: Cleanup pm runtime calls]
Signed-off-by: Vivek Gautam <[email protected]>
Reviewed-by: Tomasz Figa <[email protected]>
---
drivers/iommu/arm-smmu.c | 95 ++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 87 insertions(+), 8 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index d5873d545024..56a04ae80bf3 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
{ 0, NULL},
};

+static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
+{
+ if (pm_runtime_enabled(smmu->dev))
+ return pm_runtime_get_sync(smmu->dev);
+
+ return 0;
+}
+
+static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
+{
+ if (pm_runtime_enabled(smmu->dev))
+ pm_runtime_put(smmu->dev);
+}
+
static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
{
return container_of(dom, struct arm_smmu_domain, domain);
@@ -913,11 +927,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_device *smmu = smmu_domain->smmu;
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- int irq;
+ int ret, irq;

if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
return;

+ ret = arm_smmu_rpm_get(smmu);
+ if (ret < 0)
+ return;
+
/*
* Disable the context bank and free the page tables before freeing
* it.
@@ -932,6 +950,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)

free_io_pgtable_ops(smmu_domain->pgtbl_ops);
__arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
+
+ arm_smmu_rpm_put(smmu);
}

static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
@@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return -ENODEV;

smmu = fwspec_smmu(fwspec);
+
+ ret = arm_smmu_rpm_get(smmu);
+ if (ret < 0)
+ return ret;
+
/* Ensure that the domain is finalised */
ret = arm_smmu_init_domain_context(domain, smmu);
if (ret < 0)
- return ret;
+ goto rpm_put;

/*
* Sanity check the domain. We don't support domains across
@@ -1230,29 +1255,47 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
}

/* Looks ok, so add the device to the domain */
- return arm_smmu_domain_add_master(smmu_domain, fwspec);
+ ret = arm_smmu_domain_add_master(smmu_domain, fwspec);
+
+rpm_put:
+ arm_smmu_rpm_put(smmu);
+ return ret;
}

static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot)
{
struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ int ret;

if (!ops)
return -ENODEV;

- return ops->map(ops, iova, paddr, size, prot);
+ arm_smmu_rpm_get(smmu);
+ ret = ops->map(ops, iova, paddr, size, prot);
+ arm_smmu_rpm_put(smmu);
+
+ return ret;
}

static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
size_t size)
{
struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ size_t ret;

if (!ops)
return 0;

- return ops->unmap(ops, iova, size);
+ arm_smmu_rpm_get(smmu);
+ ret = ops->unmap(ops, iova, size);
+ arm_smmu_rpm_put(smmu);
+
+ return ret;
}

static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
@@ -1407,14 +1450,22 @@ static int arm_smmu_add_device(struct device *dev)
while (i--)
cfg->smendx[i] = INVALID_SMENDX;

+ ret = arm_smmu_rpm_get(smmu);
+ if (ret < 0)
+ goto out_cfg_free;
+
ret = arm_smmu_master_alloc_smes(dev);
if (ret)
- goto out_cfg_free;
+ goto out_rpm_put;

iommu_device_link(&smmu->iommu, dev);

+ arm_smmu_rpm_put(smmu);
+
return 0;

+out_rpm_put:
+ arm_smmu_rpm_put(smmu);
out_cfg_free:
kfree(cfg);
out_free:
@@ -1427,7 +1478,7 @@ static void arm_smmu_remove_device(struct device *dev)
struct iommu_fwspec *fwspec = dev->iommu_fwspec;
struct arm_smmu_master_cfg *cfg;
struct arm_smmu_device *smmu;
-
+ int ret;

if (!fwspec || fwspec->ops != &arm_smmu_ops)
return;
@@ -1435,8 +1486,15 @@ static void arm_smmu_remove_device(struct device *dev)
cfg = fwspec->iommu_priv;
smmu = cfg->smmu;

+ ret = arm_smmu_rpm_get(smmu);
+ if (ret < 0)
+ return;
+
iommu_device_unlink(&smmu->iommu, dev);
arm_smmu_master_free_smes(fwspec);
+
+ arm_smmu_rpm_put(smmu);
+
iommu_group_remove_device(dev);
kfree(fwspec->iommu_priv);
iommu_fwspec_free(dev);
@@ -2124,6 +2182,8 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
smmu->irqs[i] = irq;
}

+ platform_set_drvdata(pdev, smmu);
+
err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
if (err)
return err;
@@ -2132,6 +2192,19 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
if (err)
return err;

+ /*
+ * We want to avoid touching dev->power.lock in fastpaths unless
+ * it's really going to do something useful - pm_runtime_enabled()
+ * can serve as an ideal proxy for that decision. So, conditionally
+ * enable pm_runtime.
+ */
+ if (dev->pm_domain)
+ pm_runtime_enable(dev);
+
+ err = arm_smmu_rpm_get(smmu);
+ if (err < 0)
+ return err;
+
err = arm_smmu_device_cfg_probe(smmu);
if (err)
return err;
@@ -2173,10 +2246,11 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
return err;
}

- platform_set_drvdata(pdev, smmu);
arm_smmu_device_reset(smmu);
arm_smmu_test_smr_masks(smmu);

+ arm_smmu_rpm_put(smmu);
+
/*
* For ACPI and generic DT bindings, an SMMU will be probed before
* any device which might need it, so we want the bus ops in place
@@ -2212,8 +2286,13 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
dev_err(&pdev->dev, "removing device with active domains!\n");

+ arm_smmu_rpm_get(smmu);
/* Turn the thing off */
writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+ arm_smmu_rpm_put(smmu);
+
+ if (pm_runtime_enabled(smmu->dev))
+ pm_runtime_disable(smmu->dev);

clk_bulk_unprepare(smmu->num_clks, smmu->clks);

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 08:59:01

[permalink] [raw]

Subject: [PATCH v9 1/5] driver core: Find an existing link between two devices

The lists managing the device-links can be traversed to
find the link between two devices. The device_link_add() APIs
does traverse these lists to check if there's already a link
setup between the two devices.
So, add a new APIs, device_link_find(), to find an existing
device link between two devices - suppliers and consumers.

Signed-off-by: Vivek Gautam <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
---

* New patch added to this series.

drivers/base/core.c | 30 +++++++++++++++++++++++++++---
include/linux/device.h | 2 ++
2 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index 5847364f25d9..e8c9774e4ba2 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -144,6 +144,30 @@ static int device_reorder_to_tail(struct device *dev, void *not_used)
return 0;
}

+/**
+ * device_link_find - find any existing link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ *
+ * Returns pointer to the existing link between a supplier and
+ * and consumer devices, or NULL if no link exists.
+ */
+struct device_link *device_link_find(struct device *consumer,
+ struct device *supplier)
+{
+ struct device_link *link = NULL;
+
+ if (!consumer || !supplier)
+ return NULL;
+
+ list_for_each_entry(link, &supplier->links.consumers, s_node)
+ if (link->consumer == consumer)
+ break;
+
+ return link;
+}
+EXPORT_SYMBOL_GPL(device_link_find);
+
/**
* device_link_add - Create a link between two devices.
* @consumer: Consumer end of the link.
@@ -195,9 +219,9 @@ struct device_link *device_link_add(struct device *consumer,
goto out;
}

- list_for_each_entry(link, &supplier->links.consumers, s_node)
- if (link->consumer == consumer)
- goto out;
+ link = device_link_find(consumer, supplier);
+ if (link)
+ goto out;

link = kzalloc(sizeof(*link), GFP_KERNEL);
if (!link)
diff --git a/include/linux/device.h b/include/linux/device.h
index b093405ed525..13bc1884c3eb 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -1278,6 +1278,8 @@ extern const char *dev_driver_string(const struct device *dev);
struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32 flags);
void device_link_del(struct device_link *link);
+struct device_link *device_link_find(struct device *consumer,
+ struct device *supplier);

#ifdef CONFIG_PRINTK

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 09:41:35

by Rafael J. Wysocki

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Tuesday, March 13, 2018 9:55:30 AM CET Vivek Gautam wrote:
> The lists managing the device-links can be traversed to
> find the link between two devices. The device_link_add() APIs
> does traverse these lists to check if there's already a link
> setup between the two devices.
> So, add a new APIs, device_link_find(), to find an existing
> device link between two devices - suppliers and consumers.
>
> Signed-off-by: Vivek Gautam <[email protected]>
> Cc: Rafael J. Wysocki <[email protected]>
> Cc: Greg Kroah-Hartman <[email protected]>
> ---
>
> * New patch added to this series.
>
> drivers/base/core.c | 30 +++++++++++++++++++++++++++---
> include/linux/device.h | 2 ++
> 2 files changed, 29 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index 5847364f25d9..e8c9774e4ba2 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -144,6 +144,30 @@ static int device_reorder_to_tail(struct device *dev, void *not_used)
> return 0;
> }
>
> +/**
> + * device_link_find - find any existing link between two devices.
> + * @consumer: Consumer end of the link.
> + * @supplier: Supplier end of the link.
> + *
> + * Returns pointer to the existing link between a supplier and
> + * and consumer devices, or NULL if no link exists.
> + */
> +struct device_link *device_link_find(struct device *consumer,
> + struct device *supplier)
> +{
> + struct device_link *link = NULL;
> +
> + if (!consumer || !supplier)
> + return NULL;
> +
> + list_for_each_entry(link, &supplier->links.consumers, s_node)
> + if (link->consumer == consumer)
> + break;
> +

Any mutual exclusion?

Or is the caller expected to take care of it? And if so, then how?

> + return link;
> +}
> +EXPORT_SYMBOL_GPL(device_link_find);
> +
> /**
> * device_link_add - Create a link between two devices.
> * @consumer: Consumer end of the link.
> @@ -195,9 +219,9 @@ struct device_link *device_link_add(struct device *consumer,
> goto out;
> }
>
> - list_for_each_entry(link, &supplier->links.consumers, s_node)
> - if (link->consumer == consumer)
> - goto out;
> + link = device_link_find(consumer, supplier);
> + if (link)
> + goto out;
>
> link = kzalloc(sizeof(*link), GFP_KERNEL);
> if (!link)
> diff --git a/include/linux/device.h b/include/linux/device.h
> index b093405ed525..13bc1884c3eb 100644
> --- a/include/linux/device.h
> +++ b/include/linux/device.h
> @@ -1278,6 +1278,8 @@ extern const char *dev_driver_string(const struct device *dev);
> struct device_link *device_link_add(struct device *consumer,
> struct device *supplier, u32 flags);
> void device_link_del(struct device_link *link);
> +struct device_link *device_link_find(struct device *consumer,
> + struct device *supplier);
>
> #ifdef CONFIG_PRINTK
>
>

2018-03-13 09:56:42

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Tue, Mar 13, 2018 at 3:10 PM, Rafael J. Wysocki <[email protected]> wrote:
> On Tuesday, March 13, 2018 9:55:30 AM CET Vivek Gautam wrote:
>> The lists managing the device-links can be traversed to
>> find the link between two devices. The device_link_add() APIs
>> does traverse these lists to check if there's already a link
>> setup between the two devices.
>> So, add a new APIs, device_link_find(), to find an existing
>> device link between two devices - suppliers and consumers.
>>
>> Signed-off-by: Vivek Gautam <[email protected]>
>> Cc: Rafael J. Wysocki <[email protected]>
>> Cc: Greg Kroah-Hartman <[email protected]>
>> ---
>>
>> * New patch added to this series.
>>
>> drivers/base/core.c | 30 +++++++++++++++++++++++++++---
>> include/linux/device.h | 2 ++
>> 2 files changed, 29 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/base/core.c b/drivers/base/core.c
>> index 5847364f25d9..e8c9774e4ba2 100644
>> --- a/drivers/base/core.c
>> +++ b/drivers/base/core.c
>> @@ -144,6 +144,30 @@ static int device_reorder_to_tail(struct device *dev, void *not_used)
>> return 0;
>> }
>>
>> +/**
>> + * device_link_find - find any existing link between two devices.
>> + * @consumer: Consumer end of the link.
>> + * @supplier: Supplier end of the link.
>> + *
>> + * Returns pointer to the existing link between a supplier and
>> + * and consumer devices, or NULL if no link exists.
>> + */
>> +struct device_link *device_link_find(struct device *consumer,
>> + struct device *supplier)
>> +{
>> + struct device_link *link = NULL;
>> +
>> + if (!consumer || !supplier)
>> + return NULL;
>> +
>> + list_for_each_entry(link, &supplier->links.consumers, s_node)
>> + if (link->consumer == consumer)
>> + break;
>> +
>
> Any mutual exclusion?
>
> Or is the caller expected to take care of it? And if so, then how?

I think it's better that we take care of lock here in the code rather
than depending
on the caller.
But i can't take device_links_write_lock() since device_link_add()
already takes that.

regards
Vivek

>
>> + return link;
>> +}
>> +EXPORT_SYMBOL_GPL(device_link_find);
>> +
>> /**
>> * device_link_add - Create a link between two devices.
>> * @consumer: Consumer end of the link.
>> @@ -195,9 +219,9 @@ struct device_link *device_link_add(struct device *consumer,
>> goto out;
>> }
>>
>> - list_for_each_entry(link, &supplier->links.consumers, s_node)
>> - if (link->consumer == consumer)
>> - goto out;
>> + link = device_link_find(consumer, supplier);
>> + if (link)
>> + goto out;
>>
>> link = kzalloc(sizeof(*link), GFP_KERNEL);
>> if (!link)
>> diff --git a/include/linux/device.h b/include/linux/device.h
>> index b093405ed525..13bc1884c3eb 100644
>> --- a/include/linux/device.h
>> +++ b/include/linux/device.h
>> @@ -1278,6 +1278,8 @@ extern const char *dev_driver_string(const struct device *dev);
>> struct device_link *device_link_add(struct device *consumer,
>> struct device *supplier, u32 flags);
>> void device_link_del(struct device_link *link);
>> +struct device_link *device_link_find(struct device *consumer,
>> + struct device *supplier);
>>
>> #ifdef CONFIG_PRINTK
>>
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 10:00:33

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Tue, Mar 13, 2018 at 2:25 PM, Vivek Gautam
<[email protected]> wrote:
> The lists managing the device-links can be traversed to
> find the link between two devices. The device_link_add() APIs
> does traverse these lists to check if there's already a link
> setup between the two devices.
> So, add a new APIs, device_link_find(), to find an existing
> device link between two devices - suppliers and consumers.
>
> Signed-off-by: Vivek Gautam <[email protected]>
> Cc: Rafael J. Wysocki <[email protected]>
> Cc: Greg Kroah-Hartman <[email protected]>
> ---
>
> * New patch added to this series.
>
> drivers/base/core.c | 30 +++++++++++++++++++++++++++---
> include/linux/device.h | 2 ++
> 2 files changed, 29 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index 5847364f25d9..e8c9774e4ba2 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -144,6 +144,30 @@ static int device_reorder_to_tail(struct device *dev, void *not_used)
> return 0;
> }
>
> +/**
> + * device_link_find - find any existing link between two devices.
> + * @consumer: Consumer end of the link.
> + * @supplier: Supplier end of the link.
> + *
> + * Returns pointer to the existing link between a supplier and
> + * and consumer devices, or NULL if no link exists.
> + */
> +struct device_link *device_link_find(struct device *consumer,
> + struct device *supplier)
> +{
> + struct device_link *link = NULL;
> +
> + if (!consumer || !supplier)
> + return NULL;
> +
> + list_for_each_entry(link, &supplier->links.consumers, s_node)
> + if (link->consumer == consumer)
> + break;
> +
> + return link;

My bad, this too needs fixing (didn't add the changes to the patch :( )

list_for_each_entry(link, &supplier->links.consumers, s_node)
if (link->consumer == consumer)
return link;

return NULL;

> +}
> +EXPORT_SYMBOL_GPL(device_link_find);
> +
> /**
> * device_link_add - Create a link between two devices.
> * @consumer: Consumer end of the link.
> @@ -195,9 +219,9 @@ struct device_link *device_link_add(struct device *consumer,
> goto out;
> }
>
> - list_for_each_entry(link, &supplier->links.consumers, s_node)
> - if (link->consumer == consumer)
> - goto out;
> + link = device_link_find(consumer, supplier);
> + if (link)
> + goto out;
>
> link = kzalloc(sizeof(*link), GFP_KERNEL);
> if (!link)
> diff --git a/include/linux/device.h b/include/linux/device.h
> index b093405ed525..13bc1884c3eb 100644
> --- a/include/linux/device.h
> +++ b/include/linux/device.h
> @@ -1278,6 +1278,8 @@ extern const char *dev_driver_string(const struct device *dev);
> struct device_link *device_link_add(struct device *consumer,
> struct device *supplier, u32 flags);
> void device_link_del(struct device_link *link);
> +struct device_link *device_link_find(struct device *consumer,
> + struct device *supplier);
>
> #ifdef CONFIG_PRINTK
>
> --
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
> of Code Aurora Forum, hosted by The Linux Foundation
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 10:17:23

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

Hi Vivek,

Thanks for the patch.

On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
<[email protected]> wrote:
> The lists managing the device-links can be traversed to
> find the link between two devices. The device_link_add() APIs
> does traverse these lists to check if there's already a link
> setup between the two devices.
> So, add a new APIs, device_link_find(), to find an existing
> device link between two devices - suppliers and consumers.

I'm wondering if this API would be useful for anything else that the
problem we're trying to solve with deleting links without storing them
anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
better alternative?

Best regards,
Tomasz

2018-03-13 10:35:43

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

Hi Tomasz,

On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
> Hi Vivek,
>
> Thanks for the patch.
>
> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
> <[email protected]> wrote:
>> The lists managing the device-links can be traversed to
>> find the link between two devices. The device_link_add() APIs
>> does traverse these lists to check if there's already a link
>> setup between the two devices.
>> So, add a new APIs, device_link_find(), to find an existing
>> device link between two devices - suppliers and consumers.
>
> I'm wondering if this API would be useful for anything else that the
> problem we're trying to solve with deleting links without storing them
> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
> better alternative?

Yea, that sounds simpler i think. Will add this API instead of
find_link(). Thanks.

regards
vivek
>
> Best regards,
> Tomasz
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-13 11:25:05

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam
<[email protected]> wrote:
> Hi Tomasz,
>
> On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
>> Hi Vivek,
>>
>> Thanks for the patch.
>>
>> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
>> <[email protected]> wrote:
>>> The lists managing the device-links can be traversed to
>>> find the link between two devices. The device_link_add() APIs
>>> does traverse these lists to check if there's already a link
>>> setup between the two devices.
>>> So, add a new APIs, device_link_find(), to find an existing
>>> device link between two devices - suppliers and consumers.
>>
>> I'm wondering if this API would be useful for anything else that the
>> problem we're trying to solve with deleting links without storing them
>> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
>> better alternative?
>
> Yea, that sounds simpler i think. Will add this API instead of
> find_link(). Thanks.

Perhaps let's wait for a moment to see if there are other opinions. :)

Rafael, Lucas, any thoughts?

Best regards,
Tomasz

2018-03-13 12:51:00

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On 13/03/18 09:55, Vivek Gautam wrote:
> On Tue, Mar 13, 2018 at 3:10 PM, Rafael J. Wysocki <[email protected]> wrote:
>> On Tuesday, March 13, 2018 9:55:30 AM CET Vivek Gautam wrote:
>>> The lists managing the device-links can be traversed to
>>> find the link between two devices. The device_link_add() APIs
>>> does traverse these lists to check if there's already a link
>>> setup between the two devices.
>>> So, add a new APIs, device_link_find(), to find an existing
>>> device link between two devices - suppliers and consumers.
>>>
>>> Signed-off-by: Vivek Gautam <[email protected]>
>>> Cc: Rafael J. Wysocki <[email protected]>
>>> Cc: Greg Kroah-Hartman <[email protected]>
>>> ---
>>>
>>> * New patch added to this series.
>>>
>>> drivers/base/core.c | 30 +++++++++++++++++++++++++++---
>>> include/linux/device.h | 2 ++
>>> 2 files changed, 29 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/base/core.c b/drivers/base/core.c
>>> index 5847364f25d9..e8c9774e4ba2 100644
>>> --- a/drivers/base/core.c
>>> +++ b/drivers/base/core.c
>>> @@ -144,6 +144,30 @@ static int device_reorder_to_tail(struct device *dev, void *not_used)
>>> return 0;
>>> }
>>>
>>> +/**
>>> + * device_link_find - find any existing link between two devices.
>>> + * @consumer: Consumer end of the link.
>>> + * @supplier: Supplier end of the link.
>>> + *
>>> + * Returns pointer to the existing link between a supplier and
>>> + * and consumer devices, or NULL if no link exists.
>>> + */
>>> +struct device_link *device_link_find(struct device *consumer,
>>> + struct device *supplier)
>>> +{
>>> + struct device_link *link = NULL;
>>> +
>>> + if (!consumer || !supplier)
>>> + return NULL;
>>> +
>>> + list_for_each_entry(link, &supplier->links.consumers, s_node)
>>> + if (link->consumer == consumer)
>>> + break;
>>> +
>>
>> Any mutual exclusion?
>>
>> Or is the caller expected to take care of it? And if so, then how?
>
> I think it's better that we take care of lock here in the code rather
> than depending
> on the caller.
> But i can't take device_links_write_lock() since device_link_add()
> already takes that.

Well, the normal pattern is to break out the internal helper function
as-is, then add a public wrapper which validates inputs, handles
locking, etc., equivalently to existing caller(s). See what
device_link_del() and others do, e.g.:

static struct device_link *__device_link_find(struct device *consumer,
struct device *supplier)
{
list_for_each_entry(link, &supplier->links.consumers, s_node)
if (link->consumer == consumer)
return link;
return NULL;
}

struct device_link *device_link_find(struct device *consumer,
struct device *supplier)
{
struct device_link *link;

if (!consumer || !supplier)
return NULL;

device_links_write_lock();
link = __device_link_find(consumer, supplier);
device_links_write_unlock();
return link;
}

where device_link_add() would call __device_link_find() directly.

However, as Tomasz points out (and I hadn't really considered), if the
only reasonable thing to with a link once you've found it is to delete
it, then in terms of the public API it may well make more sense to just
implement something like a device_link_remove() which does both in one go.

Robin.

2018-03-13 14:40:30

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

Hi Robin,

On Tue, Mar 13, 2018 at 6:19 PM, Robin Murphy <[email protected]> wrote:
> On 13/03/18 09:55, Vivek Gautam wrote:
>>
>> On Tue, Mar 13, 2018 at 3:10 PM, Rafael J. Wysocki <[email protected]>
>> wrote:
>>>
>>> On Tuesday, March 13, 2018 9:55:30 AM CET Vivek Gautam wrote:
>>>>
>>>> The lists managing the device-links can be traversed to
>>>> find the link between two devices. The device_link_add() APIs
>>>> does traverse these lists to check if there's already a link
>>>> setup between the two devices.
>>>> So, add a new APIs, device_link_find(), to find an existing
>>>> device link between two devices - suppliers and consumers.
>>>>
>>>> Signed-off-by: Vivek Gautam <[email protected]>
>>>> Cc: Rafael J. Wysocki <[email protected]>
>>>> Cc: Greg Kroah-Hartman <[email protected]>
>>>> ---
>>>>
>>>> * New patch added to this series.
>>>>
>>>> drivers/base/core.c | 30 +++++++++++++++++++++++++++---
>>>> include/linux/device.h | 2 ++
>>>> 2 files changed, 29 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/drivers/base/core.c b/drivers/base/core.c
>>>> index 5847364f25d9..e8c9774e4ba2 100644
>>>> --- a/drivers/base/core.c
>>>> +++ b/drivers/base/core.c
>>>> @@ -144,6 +144,30 @@ static int device_reorder_to_tail(struct device
>>>> *dev, void *not_used)
>>>> return 0;
>>>> }
>>>>
>>>> +/**
>>>> + * device_link_find - find any existing link between two devices.
>>>> + * @consumer: Consumer end of the link.
>>>> + * @supplier: Supplier end of the link.
>>>> + *
>>>> + * Returns pointer to the existing link between a supplier and
>>>> + * and consumer devices, or NULL if no link exists.
>>>> + */
>>>> +struct device_link *device_link_find(struct device *consumer,
>>>> + struct device *supplier)
>>>> +{
>>>> + struct device_link *link = NULL;
>>>> +
>>>> + if (!consumer || !supplier)
>>>> + return NULL;
>>>> +
>>>> + list_for_each_entry(link, &supplier->links.consumers, s_node)
>>>> + if (link->consumer == consumer)
>>>> + break;
>>>> +
>>>
>>>
>>> Any mutual exclusion?
>>>
>>> Or is the caller expected to take care of it? And if so, then how?
>>
>>
>> I think it's better that we take care of lock here in the code rather
>> than depending
>> on the caller.
>> But i can't take device_links_write_lock() since device_link_add()
>> already takes that.
>
>
> Well, the normal pattern is to break out the internal helper function as-is,
> then add a public wrapper which validates inputs, handles locking, etc.,
> equivalently to existing caller(s). See what device_link_del() and others
> do, e.g.:
>
> static struct device_link *__device_link_find(struct device *consumer,
> struct device *supplier)
> {
> list_for_each_entry(link, &supplier->links.consumers, s_node)
> if (link->consumer == consumer)
> return link;
> return NULL;
> }
>
> struct device_link *device_link_find(struct device *consumer,
> struct device *supplier)
> {
> struct device_link *link;
>
> if (!consumer || !supplier)
> return NULL;
>
> device_links_write_lock();
> link = __device_link_find(consumer, supplier);
> device_links_write_unlock();
> return link;
> }
>
> where device_link_add() would call __device_link_find() directly.

Right, I understand it now. Thanks for detailed explanation.

regards
Vivek

>
> However, as Tomasz points out (and I hadn't really considered), if the only
> reasonable thing to with a link once you've found it is to delete it, then
> in terms of the public API it may well make more sense to just implement
> something like a device_link_remove() which does both in one go.
>
> Robin.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-14 11:12:39

by Rafael J. Wysocki

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
> On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam
> <[email protected]> wrote:
> > Hi Tomasz,
> >
> > On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
> >> Hi Vivek,
> >>
> >> Thanks for the patch.
> >>
> >> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
> >> <[email protected]> wrote:
> >>> The lists managing the device-links can be traversed to
> >>> find the link between two devices. The device_link_add() APIs
> >>> does traverse these lists to check if there's already a link
> >>> setup between the two devices.
> >>> So, add a new APIs, device_link_find(), to find an existing
> >>> device link between two devices - suppliers and consumers.
> >>
> >> I'm wondering if this API would be useful for anything else that the
> >> problem we're trying to solve with deleting links without storing them
> >> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
> >> better alternative?
> >
> > Yea, that sounds simpler i think. Will add this API instead of
> > find_link(). Thanks.
>
> Perhaps let's wait for a moment to see if there are other opinions. :)
>
> Rafael, Lucas, any thoughts?

It is not clear to me what the device_link_del_dev(consumer, supplier) would do.

Thanks,
Rafael

2018-03-14 11:58:06

by Rafael J. Wysocki

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Wednesday, March 14, 2018 12:50:54 PM CET Tomasz Figa wrote:
> On Wed, Mar 14, 2018 at 8:12 PM, Rafael J. Wysocki <[email protected]> wrote:
> > On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
> >> On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam
> >> <[email protected]> wrote:
> >> > Hi Tomasz,
> >> >
> >> > On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
> >> >> Hi Vivek,
> >> >>
> >> >> Thanks for the patch.
> >> >>
> >> >> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
> >> >> <[email protected]> wrote:
> >> >>> The lists managing the device-links can be traversed to
> >> >>> find the link between two devices. The device_link_add() APIs
> >> >>> does traverse these lists to check if there's already a link
> >> >>> setup between the two devices.
> >> >>> So, add a new APIs, device_link_find(), to find an existing
> >> >>> device link between two devices - suppliers and consumers.
> >> >>
> >> >> I'm wondering if this API would be useful for anything else that the
> >> >> problem we're trying to solve with deleting links without storing them
> >> >> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
> >> >> better alternative?
> >> >
> >> > Yea, that sounds simpler i think. Will add this API instead of
> >> > find_link(). Thanks.
> >>
> >> Perhaps let's wait for a moment to see if there are other opinions. :)
> >>
> >> Rafael, Lucas, any thoughts?
> >
> > It is not clear to me what the device_link_del_dev(consumer, supplier) would do.
>
> It would delete a link between consumer and supplier.

If there's one I suppose.

I'm wondering if you are somehow trying to address the same problem as the
device links reference counting patch from Lukas that has been queued up for 4.17
already.

Thanks,
Rafael

2018-03-14 11:59:58

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Wed, Mar 14, 2018 at 8:12 PM, Rafael J. Wysocki <[email protected]> wrote:
> On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
>> On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam
>> <[email protected]> wrote:
>> > Hi Tomasz,
>> >
>> > On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
>> >> Hi Vivek,
>> >>
>> >> Thanks for the patch.
>> >>
>> >> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
>> >> <[email protected]> wrote:
>> >>> The lists managing the device-links can be traversed to
>> >>> find the link between two devices. The device_link_add() APIs
>> >>> does traverse these lists to check if there's already a link
>> >>> setup between the two devices.
>> >>> So, add a new APIs, device_link_find(), to find an existing
>> >>> device link between two devices - suppliers and consumers.
>> >>
>> >> I'm wondering if this API would be useful for anything else that the
>> >> problem we're trying to solve with deleting links without storing them
>> >> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
>> >> better alternative?
>> >
>> > Yea, that sounds simpler i think. Will add this API instead of
>> > find_link(). Thanks.
>>
>> Perhaps let's wait for a moment to see if there are other opinions. :)
>>
>> Rafael, Lucas, any thoughts?
>
> It is not clear to me what the device_link_del_dev(consumer, supplier) would do.

It would delete a link between consumer and supplier.

Best regards,
Tomasz

2018-03-14 12:16:29

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

Hi Rafael,

On 14/03/18 11:57, Rafael J. Wysocki wrote:
> On Wednesday, March 14, 2018 12:50:54 PM CET Tomasz Figa wrote:
>> On Wed, Mar 14, 2018 at 8:12 PM, Rafael J. Wysocki <[email protected]> wrote:
>>> On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
>>>> On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam
>>>> <[email protected]> wrote:
>>>>> Hi Tomasz,
>>>>>
>>>>> On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
>>>>>> Hi Vivek,
>>>>>>
>>>>>> Thanks for the patch.
>>>>>>
>>>>>> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam
>>>>>> <[email protected]> wrote:
>>>>>>> The lists managing the device-links can be traversed to
>>>>>>> find the link between two devices. The device_link_add() APIs
>>>>>>> does traverse these lists to check if there's already a link
>>>>>>> setup between the two devices.
>>>>>>> So, add a new APIs, device_link_find(), to find an existing
>>>>>>> device link between two devices - suppliers and consumers.
>>>>>>
>>>>>> I'm wondering if this API would be useful for anything else that the
>>>>>> problem we're trying to solve with deleting links without storing them
>>>>>> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
>>>>>> better alternative?
>>>>>
>>>>> Yea, that sounds simpler i think. Will add this API instead of
>>>>> find_link(). Thanks.
>>>>
>>>> Perhaps let's wait for a moment to see if there are other opinions. :)
>>>>
>>>> Rafael, Lucas, any thoughts?
>>>
>>> It is not clear to me what the device_link_del_dev(consumer, supplier) would do.
>>
>> It would delete a link between consumer and supplier.
>
> If there's one I suppose.
>
> I'm wondering if you are somehow trying to address the same problem as the
> device links reference counting patch from Lukas that has been queued up for 4.17
> already.

Not quite - the issue here is that we have one supplier with an
arbitrarily large number of consumers, and would prefer that supplier
not to have to spend a whole bunch of memory to store all the struct
device_link pointers for the sole reason of having something to give to
device_link_del() at the end, given that the device links code is
already keeping track of everything internally anyway.

The current API would permit doing this:

iommu_attach(dev) {
...
if (!device_link_add(dev, iommu, IOMMU_LINK_FLAGS))
return -ENODEV;
...
}

iommu_detach(dev) {
...
// Will return the existing link from earlier
link = device_link_add(dev, iommu, IOMMU_LINK_FLAGS);
device_link_del(link);
// Needed once refcounting is in place
//device_link_del(link);
...
}

but it looks so wacky and non-obvious that we'd like to encapsulate the
same behaviour into a more formal interface (my personal naming
preference would be device_link_remove(consumer, supplier)).

Robin.

2018-03-14 12:24:42

by Lukas Wunner

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Wed, Mar 14, 2018 at 12:12:05PM +0100, Rafael J. Wysocki wrote:
> On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
> > On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam <[email protected]> wrote:
> > > On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
> > >> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam <[email protected]> wrote:
> > >>> The lists managing the device-links can be traversed to
> > >>> find the link between two devices. The device_link_add() APIs
> > >>> does traverse these lists to check if there's already a link
> > >>> setup between the two devices.
> > >>> So, add a new APIs, device_link_find(), to find an existing
> > >>> device link between two devices - suppliers and consumers.
> > >>
> > >> I'm wondering if this API would be useful for anything else that the
> > >> problem we're trying to solve with deleting links without storing them
> > >> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
> > >> better alternative?
> > >
> > > Yea, that sounds simpler i think. Will add this API instead of
> > > find_link(). Thanks.
> >
> > Perhaps let's wait for a moment to see if there are other opinions. :)
> >
> > Rafael, Lucas, any thoughts?
>
> It is not clear to me what the device_link_del_dev(consumer, supplier)
> would do.

The point appears to be that the pointer to the device_link need not be
stored somewhere for later deletion. The newly added function would
check if a device link exists and delete it if so.

However I don't understand why storing the pointer would be a problem?
Also, would using DL_FLAG_AUTOREMOVE avoid the need for the additional
function?

Thanks,

Lukas

2018-03-14 12:30:49

by Lukas Wunner

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

On Wed, Mar 14, 2018 at 12:14:15PM +0000, Robin Murphy wrote:
> >>On Wed, Mar 14, 2018 at 8:12 PM, Rafael J. Wysocki <[email protected]> wrote:
> >>>On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
> >>>>On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam <[email protected]> wrote:
> >>>>>On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
> >>>>>>On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam <[email protected]> wrote:
> >>>>>>>The lists managing the device-links can be traversed to
> >>>>>>>find the link between two devices. The device_link_add() APIs
> >>>>>>>does traverse these lists to check if there's already a link
> >>>>>>>setup between the two devices.
> >>>>>>>So, add a new APIs, device_link_find(), to find an existing
> >>>>>>>device link between two devices - suppliers and consumers.
> >>>>>>
> >>>>>>I'm wondering if this API would be useful for anything else that the
> >>>>>>problem we're trying to solve with deleting links without storing them
> >>>>>>anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
> >>>>>>better alternative?
> >>>>>
> >>>>>Yea, that sounds simpler i think. Will add this API instead of
> >>>>>find_link(). Thanks.
> >>>>
> >>>>Perhaps let's wait for a moment to see if there are other opinions. :)
> >>>>
> >>>>Rafael, Lucas, any thoughts?
> >>>
> >>>It is not clear to me what the device_link_del_dev(consumer, supplier)
> >>>would do.
>
> Not quite - the issue here is that we have one supplier with an arbitrarily
> large number of consumers, and would prefer that supplier not to have to
> spend a whole bunch of memory to store all the struct device_link pointers
> for the sole reason of having something to give to device_link_del() at the
> end, given that the device links code is already keeping track of everything
> internally anyway.

Makes sense to me. How about an additional flag which autoremoves the
link on provider unbind?

Thanks,

Lukas

2018-03-14 17:47:21

[permalink] [raw]

Subject: Re: [PATCH v9 3/5] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

On 13/03/18 08:55, Vivek Gautam wrote:
> From: Sricharan R <[email protected]>
>
> The smmu device probe/remove and add/remove master device callbacks
> gets called when the smmu is not linked to its master, that is without
> the context of the master device. So calling runtime apis in those places
> separately.
>
> Signed-off-by: Sricharan R <[email protected]>
> [vivek: Cleanup pm runtime calls]
> Signed-off-by: Vivek Gautam <[email protected]>
> Reviewed-by: Tomasz Figa <[email protected]>
> ---
> drivers/iommu/arm-smmu.c | 95 ++++++++++++++++++++++++++++++++++++++++++++----
> 1 file changed, 87 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index d5873d545024..56a04ae80bf3 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
> { 0, NULL},
> };
>
> +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
> +{
> + if (pm_runtime_enabled(smmu->dev))
> + return pm_runtime_get_sync(smmu->dev);
> +
> + return 0;
> +}
> +
> +static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
> +{
> + if (pm_runtime_enabled(smmu->dev))
> + pm_runtime_put(smmu->dev);
> +}
> +
> static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> {
> return container_of(dom, struct arm_smmu_domain, domain);
> @@ -913,11 +927,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_device *smmu = smmu_domain->smmu;
> struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> - int irq;
> + int ret, irq;
>
> if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
> return;
>
> + ret = arm_smmu_rpm_get(smmu);
> + if (ret < 0)
> + return;
> +
> /*
> * Disable the context bank and free the page tables before freeing
> * it.
> @@ -932,6 +950,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>
> free_io_pgtable_ops(smmu_domain->pgtbl_ops);
> __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
> +
> + arm_smmu_rpm_put(smmu);
> }
>
> static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> @@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return -ENODEV;
>
> smmu = fwspec_smmu(fwspec);
> +
> + ret = arm_smmu_rpm_get(smmu);
> + if (ret < 0)
> + return ret;
> +
> /* Ensure that the domain is finalised */
> ret = arm_smmu_init_domain_context(domain, smmu);
> if (ret < 0)
> - return ret;
> + goto rpm_put;
>
> /*
> * Sanity check the domain. We don't support domains across
> @@ -1230,29 +1255,47 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> }
>
> /* Looks ok, so add the device to the domain */
> - return arm_smmu_domain_add_master(smmu_domain, fwspec);
> + ret = arm_smmu_domain_add_master(smmu_domain, fwspec);
> +
> +rpm_put:
> + arm_smmu_rpm_put(smmu);
> + return ret;
> }
>
> static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
> phys_addr_t paddr, size_t size, int prot)
> {
> struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + struct arm_smmu_device *smmu = smmu_domain->smmu;

Nit: please use arm_smmu_domain for ops as well (as it was before
523d7423e21b), or consistently elide it for smmu - the mixture of both
methods is just a horrible mess (here and in unmap).

> + int ret;
>
> if (!ops)
> return -ENODEV;
>
> - return ops->map(ops, iova, paddr, size, prot);
> + arm_smmu_rpm_get(smmu);
> + ret = ops->map(ops, iova, paddr, size, prot);
> + arm_smmu_rpm_put(smmu);
> +
> + return ret;
> }
>
> static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
> size_t size)
> {
> struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + struct arm_smmu_device *smmu = smmu_domain->smmu;
> + size_t ret;
>
> if (!ops)
> return 0;
>
> - return ops->unmap(ops, iova, size);
> + arm_smmu_rpm_get(smmu);
> + ret = ops->unmap(ops, iova, size);
> + arm_smmu_rpm_put(smmu);
> +
> + return ret;
> }
>
> static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
> @@ -1407,14 +1450,22 @@ static int arm_smmu_add_device(struct device *dev)
> while (i--)
> cfg->smendx[i] = INVALID_SMENDX;
>
> + ret = arm_smmu_rpm_get(smmu);
> + if (ret < 0)
> + goto out_cfg_free;
> +
> ret = arm_smmu_master_alloc_smes(dev);

Nit: it would be easier to just do the rpm_put here; then you don't need
to mess with the cleanup path.

> if (ret)
> - goto out_cfg_free;
> + goto out_rpm_put;
>
> iommu_device_link(&smmu->iommu, dev);
>
> + arm_smmu_rpm_put(smmu);
> +
> return 0;
>
> +out_rpm_put:
> + arm_smmu_rpm_put(smmu);
> out_cfg_free:
> kfree(cfg);
> out_free:
> @@ -1427,7 +1478,7 @@ static void arm_smmu_remove_device(struct device *dev)
> struct iommu_fwspec *fwspec = dev->iommu_fwspec;
> struct arm_smmu_master_cfg *cfg;
> struct arm_smmu_device *smmu;
> -
> + int ret;
>
> if (!fwspec || fwspec->ops != &arm_smmu_ops)
> return;
> @@ -1435,8 +1486,15 @@ static void arm_smmu_remove_device(struct device *dev)
> cfg = fwspec->iommu_priv;
> smmu = cfg->smmu;
>
> + ret = arm_smmu_rpm_get(smmu);
> + if (ret < 0)
> + return;
> +
> iommu_device_unlink(&smmu->iommu, dev);
> arm_smmu_master_free_smes(fwspec);
> +
> + arm_smmu_rpm_put(smmu);
> +
> iommu_group_remove_device(dev);
> kfree(fwspec->iommu_priv);
> iommu_fwspec_free(dev);
> @@ -2124,6 +2182,8 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
> smmu->irqs[i] = irq;
> }
>
> + platform_set_drvdata(pdev, smmu);
> +
> err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
> if (err)
> return err;
> @@ -2132,6 +2192,19 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
> if (err)
> return err;
>
> + /*
> + * We want to avoid touching dev->power.lock in fastpaths unless
> + * it's really going to do something useful - pm_runtime_enabled()
> + * can serve as an ideal proxy for that decision. So, conditionally
> + * enable pm_runtime.
> + */
> + if (dev->pm_domain)
> + pm_runtime_enable(dev);
> +
> + err = arm_smmu_rpm_get(smmu);
> + if (err < 0)
> + return err;
> +
> err = arm_smmu_device_cfg_probe(smmu);
> if (err)
> return err;
> @@ -2173,10 +2246,11 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
> return err;
> }
>
> - platform_set_drvdata(pdev, smmu);
> arm_smmu_device_reset(smmu);
> arm_smmu_test_smr_masks(smmu);
>
> + arm_smmu_rpm_put(smmu);
> +
> /*
> * For ACPI and generic DT bindings, an SMMU will be probed before
> * any device which might need it, so we want the bus ops in place
> @@ -2212,8 +2286,13 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
> dev_err(&pdev->dev, "removing device with active domains!\n");
>
> + arm_smmu_rpm_get(smmu);
> /* Turn the thing off */
> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> + arm_smmu_rpm_put(smmu);
> +
> + if (pm_runtime_enabled(smmu->dev))
> + pm_runtime_disable(smmu->dev);
>
> clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>

I don't know how runtime and system PM interact - does the reset in
arm_smmu_pm_resume need special treatment as well, or is the device
guaranteed to be powered up at that point by other means?

Robin.

2018-03-14 17:52:16

[permalink] [raw]

Subject: Re: [PATCH v9 4/5] iommu/arm-smmu: Add the device_link between masters and smmu

On 13/03/18 08:55, Vivek Gautam wrote:
> From: Sricharan R <[email protected]>
>
> Finally add the device link between the master device and
> smmu, so that the smmu gets runtime enabled/disabled only when the
> master needs it. This is done from add_device callback which gets
> called once when the master is added to the smmu.
>
> Signed-off-by: Sricharan R <[email protected]>
> Signed-off-by: Vivek Gautam <[email protected]>
> Reviewed-by: Tomasz Figa <[email protected]>
> ---
> drivers/iommu/arm-smmu.c | 29 +++++++++++++++++++++++++++++
> 1 file changed, 29 insertions(+)
>
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index 56a04ae80bf3..64953ff2281f 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -1460,10 +1460,31 @@ static int arm_smmu_add_device(struct device *dev)
>
> iommu_device_link(&smmu->iommu, dev);
>
> + if (pm_runtime_enabled(smmu->dev)) {
> + struct device_link *link;
> +
> + /*
> + * Establish the link between smmu and master, so that the
> + * smmu gets runtime enabled/disabled as per the master's
> + * needs.
> + */
> + link = device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME);
> + if (!link) {

FWIW, given that we don't really care about link itself, I'd be quite
happy to simplify that lot down to:

if (pm_runtime_enabled(smmu_dev) &&
!device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME)) {

> + dev_warn(smmu->dev,
> + "Unable to add link to the consumer %s\n",
> + dev_name(dev));

(side note: since device_link_add() already prints a message on success,
maybe it could print its own failure message too?)

Robin.

> + ret = -ENODEV;
> + goto out_unlink;
> + }
> + }
> +
> arm_smmu_rpm_put(smmu);
>
> return 0;
>
> +out_unlink:
> + iommu_device_unlink(&smmu->iommu, dev);
> + arm_smmu_master_free_smes(fwspec);
> out_rpm_put:
> arm_smmu_rpm_put(smmu);
> out_cfg_free:
> @@ -1486,6 +1507,14 @@ static void arm_smmu_remove_device(struct device *dev)
> cfg = fwspec->iommu_priv;
> smmu = cfg->smmu;
>
> + if (pm_runtime_enabled(smmu->dev)) {
> + struct device_link *link;
> +
> + link = device_link_find(dev, smmu->dev);
> + if (link)
> + device_link_del(link);
> + }
> +
> ret = arm_smmu_rpm_get(smmu);
> if (ret < 0)
> return;
>

2018-03-15 06:25:11

[permalink] [raw]

Subject: Re: [PATCH v9 4/5] iommu/arm-smmu: Add the device_link between masters and smmu

On Thu, Mar 15, 2018 at 2:50 AM, Robin Murphy <[email protected]> wrote:
> On 13/03/18 08:55, Vivek Gautam wrote:
>>
>> From: Sricharan R <[email protected]>
>>
>> Finally add the device link between the master device and
>> smmu, so that the smmu gets runtime enabled/disabled only when the
>> master needs it. This is done from add_device callback which gets
>> called once when the master is added to the smmu.
>>
>> Signed-off-by: Sricharan R <[email protected]>
>> Signed-off-by: Vivek Gautam <[email protected]>
>> Reviewed-by: Tomasz Figa <[email protected]>
>> ---
>> drivers/iommu/arm-smmu.c | 29 +++++++++++++++++++++++++++++
>> 1 file changed, 29 insertions(+)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index 56a04ae80bf3..64953ff2281f 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -1460,10 +1460,31 @@ static int arm_smmu_add_device(struct device *dev)
>> iommu_device_link(&smmu->iommu, dev);
>> + if (pm_runtime_enabled(smmu->dev)) {
>> + struct device_link *link;
>> +
>> + /*
>> + * Establish the link between smmu and master, so that the
>> + * smmu gets runtime enabled/disabled as per the master's
>> + * needs.
>> + */
>> + link = device_link_add(dev, smmu->dev,
>> DL_FLAG_PM_RUNTIME);
>> + if (!link) {
>
>
> FWIW, given that we don't really care about link itself, I'd be quite happy
> to simplify that lot down to:
>
> if (pm_runtime_enabled(smmu_dev) &&
> !device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME)) {
>
>> + dev_warn(smmu->dev,
>> + "Unable to add link to the consumer
>> %s\n",
>> + dev_name(dev));
>
>
> (side note: since device_link_add() already prints a message on success,
> maybe it could print its own failure message too?)

I think we care whether adding the link succeeded. If it fails to be
added, we might end up with a complete system lockup on a system with
power domains.

Best regards,
Tomasz

2018-03-15 07:19:52

[permalink] [raw]

Subject: Re: [PATCH v9 3/5] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

On Thu, Mar 15, 2018 at 2:46 AM, Robin Murphy <[email protected]> wrote:
> On 13/03/18 08:55, Vivek Gautam wrote:
>>
>> From: Sricharan R <[email protected]>
>>
>> The smmu device probe/remove and add/remove master device callbacks
>> gets called when the smmu is not linked to its master, that is without
>> the context of the master device. So calling runtime apis in those places
>> separately.
>>
>> Signed-off-by: Sricharan R <[email protected]>
>> [vivek: Cleanup pm runtime calls]
>> Signed-off-by: Vivek Gautam <[email protected]>
>> Reviewed-by: Tomasz Figa <[email protected]>
>> ---
>> drivers/iommu/arm-smmu.c | 95
>> ++++++++++++++++++++++++++++++++++++++++++++----
>> 1 file changed, 87 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index d5873d545024..56a04ae80bf3 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop arm_smmu_options[]
>> = {
>> { 0, NULL},
>> };
>> +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
>> +{
>> + if (pm_runtime_enabled(smmu->dev))
>> + return pm_runtime_get_sync(smmu->dev);
>> +
>> + return 0;
>> +}
>> +
>> +static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
>> +{
>> + if (pm_runtime_enabled(smmu->dev))
>> + pm_runtime_put(smmu->dev);
>> +}
>> +
>> static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
>> {
>> return container_of(dom, struct arm_smmu_domain, domain);
>> @@ -913,11 +927,15 @@ static void arm_smmu_destroy_domain_context(struct
>> iommu_domain *domain)
>> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>> struct arm_smmu_device *smmu = smmu_domain->smmu;
>> struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>> - int irq;
>> + int ret, irq;
>> if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
>> return;
>> + ret = arm_smmu_rpm_get(smmu);
>> + if (ret < 0)
>> + return;
>> +
>> /*
>> * Disable the context bank and free the page tables before
>> freeing
>> * it.
>> @@ -932,6 +950,8 @@ static void arm_smmu_destroy_domain_context(struct
>> iommu_domain *domain)
>> free_io_pgtable_ops(smmu_domain->pgtbl_ops);
>> __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
>> +
>> + arm_smmu_rpm_put(smmu);
>> }
>> static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
>> @@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct iommu_domain
>> *domain, struct device *dev)
>> return -ENODEV;
>> smmu = fwspec_smmu(fwspec);
>> +
>> + ret = arm_smmu_rpm_get(smmu);
>> + if (ret < 0)
>> + return ret;
>> +
>> /* Ensure that the domain is finalised */
>> ret = arm_smmu_init_domain_context(domain, smmu);
>> if (ret < 0)
>> - return ret;
>> + goto rpm_put;
>> /*
>> * Sanity check the domain. We don't support domains across
>> @@ -1230,29 +1255,47 @@ static int arm_smmu_attach_dev(struct iommu_domain
>> *domain, struct device *dev)
>> }
>> /* Looks ok, so add the device to the domain */
>> - return arm_smmu_domain_add_master(smmu_domain, fwspec);
>> + ret = arm_smmu_domain_add_master(smmu_domain, fwspec);
>> +
>> +rpm_put:
>> + arm_smmu_rpm_put(smmu);
>> + return ret;
>> }
>> static int arm_smmu_map(struct iommu_domain *domain, unsigned long
>> iova,
>> phys_addr_t paddr, size_t size, int prot)
>> {
>> struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
>> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>> + struct arm_smmu_device *smmu = smmu_domain->smmu;
>
>
> Nit: please use arm_smmu_domain for ops as well (as it was before
> 523d7423e21b), or consistently elide it for smmu - the mixture of both
> methods is just a horrible mess (here and in unmap).
>
>
>> + int ret;
>> if (!ops)
>> return -ENODEV;
>> - return ops->map(ops, iova, paddr, size, prot);
>> + arm_smmu_rpm_get(smmu);
>> + ret = ops->map(ops, iova, paddr, size, prot);
>> + arm_smmu_rpm_put(smmu);
>> +
>> + return ret;
>> }
>> static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned
>> long iova,
>> size_t size)
>> {
>> struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
>> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>> + struct arm_smmu_device *smmu = smmu_domain->smmu;
>> + size_t ret;
>> if (!ops)
>> return 0;
>> - return ops->unmap(ops, iova, size);
>> + arm_smmu_rpm_get(smmu);
>> + ret = ops->unmap(ops, iova, size);
>> + arm_smmu_rpm_put(smmu);
>> +
>> + return ret;
>> }
>> static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
>> @@ -1407,14 +1450,22 @@ static int arm_smmu_add_device(struct device *dev)
>> while (i--)
>> cfg->smendx[i] = INVALID_SMENDX;
>> + ret = arm_smmu_rpm_get(smmu);
>> + if (ret < 0)
>> + goto out_cfg_free;
>> +
>> ret = arm_smmu_master_alloc_smes(dev);
>
>
> Nit: it would be easier to just do the rpm_put here; then you don't need to
> mess with the cleanup path.
>
>
>> if (ret)
>> - goto out_cfg_free;
>> + goto out_rpm_put;
>> iommu_device_link(&smmu->iommu, dev);
>> + arm_smmu_rpm_put(smmu);
>> +
>> return 0;
>> +out_rpm_put:
>> + arm_smmu_rpm_put(smmu);
>> out_cfg_free:
>> kfree(cfg);
>> out_free:
>> @@ -1427,7 +1478,7 @@ static void arm_smmu_remove_device(struct device
>> *dev)
>> struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>> struct arm_smmu_master_cfg *cfg;
>> struct arm_smmu_device *smmu;
>> -
>> + int ret;
>> if (!fwspec || fwspec->ops != &arm_smmu_ops)
>> return;
>> @@ -1435,8 +1486,15 @@ static void arm_smmu_remove_device(struct device
>> *dev)
>> cfg = fwspec->iommu_priv;
>> smmu = cfg->smmu;
>> + ret = arm_smmu_rpm_get(smmu);
>> + if (ret < 0)
>> + return;
>> +
>> iommu_device_unlink(&smmu->iommu, dev);
>> arm_smmu_master_free_smes(fwspec);
>> +
>> + arm_smmu_rpm_put(smmu);
>> +
>> iommu_group_remove_device(dev);
>> kfree(fwspec->iommu_priv);
>> iommu_fwspec_free(dev);
>> @@ -2124,6 +2182,8 @@ static int arm_smmu_device_probe(struct
>> platform_device *pdev)
>> smmu->irqs[i] = irq;
>> }
>> + platform_set_drvdata(pdev, smmu);
>> +
>> err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>> if (err)
>> return err;
>> @@ -2132,6 +2192,19 @@ static int arm_smmu_device_probe(struct
>> platform_device *pdev)
>> if (err)
>> return err;
>> + /*
>> + * We want to avoid touching dev->power.lock in fastpaths unless
>> + * it's really going to do something useful - pm_runtime_enabled()
>> + * can serve as an ideal proxy for that decision. So,
>> conditionally
>> + * enable pm_runtime.
>> + */
>> + if (dev->pm_domain)
>> + pm_runtime_enable(dev);
>> +
>> + err = arm_smmu_rpm_get(smmu);
>> + if (err < 0)
>> + return err;
>> +
>> err = arm_smmu_device_cfg_probe(smmu);
>> if (err)
>> return err;
>> @@ -2173,10 +2246,11 @@ static int arm_smmu_device_probe(struct
>> platform_device *pdev)
>> return err;
>> }
>> - platform_set_drvdata(pdev, smmu);
>> arm_smmu_device_reset(smmu);
>> arm_smmu_test_smr_masks(smmu);
>> + arm_smmu_rpm_put(smmu);
>> +
>> /*
>> * For ACPI and generic DT bindings, an SMMU will be probed before
>> * any device which might need it, so we want the bus ops in place
>> @@ -2212,8 +2286,13 @@ static int arm_smmu_device_remove(struct
>> platform_device *pdev)
>> if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
>> dev_err(&pdev->dev, "removing device with active
>> domains!\n");
>> + arm_smmu_rpm_get(smmu);
>> /* Turn the thing off */
>> writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>> + arm_smmu_rpm_put(smmu);
>> +
>> + if (pm_runtime_enabled(smmu->dev))
>> + pm_runtime_disable(smmu->dev);
>> clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>>
>
>
> I don't know how runtime and system PM interact - does the reset in
> arm_smmu_pm_resume need special treatment as well, or is the device
> guaranteed to be powered up at that point by other means?

Actually, it's quite complicated...

1) device_prepare(), prevents suspending active devices by getting a
runtime enable count [1] and then, depending on whether there is a
prepare callback that could be called for this device [2] or the
device doesn't have any PM callbacks [3], it might set the
"direct_complete" flag [4].

2) Later, when device_suspend() is called, if "direct_complete" is set
(and disabling runtime PM ends up with the device still
runtime-suspended) [5], the .suspend callback will be skipped. If
"direct_complete" is not set (or direct complete fails), the suspend
callback (if one exists) would be called regardless of runtime PM
state of the device [6].

3) During system resume, if "direct_complete" was set, the resume
callback would be completely skipped [7]. Otherwise it would be called
without any special conditions [8].

4) At the end of the whole process, device_complete() would put the
remaining reference count and potentially trigger a runtime idle and
suspend, if the device was active. [9]

Now, the behavior of what happens past 2) and before 3) is affected by
PM domain callbacks, namely prepare, suspend_noirq and resume_noirq.
For genpd, genpd_prepare() never returns a positive value, so
"direct_complete" would never happen [10]. genpd_finish_suspend()
[11], called from genpd_suspend_noirq(), attempts to cut off the
power, while genpd_resume_noirq() restore it [12], so it looks like
the power would be on during the SMMU resume callback.

[1] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1671
[2] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1688
[3] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1683
[4] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1719
[5] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1492
[6] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1506
[7] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L833
[8] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L888
[9] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/main.c#L1012
[10] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/domain.c#L1019
[11] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/domain.c#L1032
[12] https://elixir.bootlin.com/linux/latest/source/drivers/base/power/domain.c#L1085

Phew. This is still with skipped wake up capability handling, since
SMMU doesn't have such.

Best regards,
Tomasz

2018-03-15 08:58:50

[permalink] [raw]

Subject: Re: [PATCH v9 4/5] iommu/arm-smmu: Add the device_link between masters and smmu

Hi Robin,

On Wed, Mar 14, 2018 at 11:20 PM, Robin Murphy <[email protected]> wrote:
> On 13/03/18 08:55, Vivek Gautam wrote:
>>
>> From: Sricharan R <[email protected]>
>>
>> Finally add the device link between the master device and
>> smmu, so that the smmu gets runtime enabled/disabled only when the
>> master needs it. This is done from add_device callback which gets
>> called once when the master is added to the smmu.
>>
>> Signed-off-by: Sricharan R <[email protected]>
>> Signed-off-by: Vivek Gautam <[email protected]>
>> Reviewed-by: Tomasz Figa <[email protected]>
>> ---
>> drivers/iommu/arm-smmu.c | 29 +++++++++++++++++++++++++++++
>> 1 file changed, 29 insertions(+)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index 56a04ae80bf3..64953ff2281f 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -1460,10 +1460,31 @@ static int arm_smmu_add_device(struct device *dev)
>> iommu_device_link(&smmu->iommu, dev);
>> + if (pm_runtime_enabled(smmu->dev)) {
>> + struct device_link *link;
>> +
>> + /*
>> + * Establish the link between smmu and master, so that the
>> + * smmu gets runtime enabled/disabled as per the master's
>> + * needs.
>> + */
>> + link = device_link_add(dev, smmu->dev,
>> DL_FLAG_PM_RUNTIME);
>> + if (!link) {
>
>
> FWIW, given that we don't really care about link itself, I'd be quite happy
> to simplify that lot down to:
>
> if (pm_runtime_enabled(smmu_dev) &&
> !device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME)) {

Sure, will update this.

>
>> + dev_warn(smmu->dev,
>> + "Unable to add link to the consumer
>> %s\n",
>> + dev_name(dev));
>
>
> (side note: since device_link_add() already prints a message on success,
> maybe it could print its own failure message too?)

Should we make device_link that verbose - to print failure messages at
each step (there are atleast a couple where we return link as NULL),
or we can let the users handle printing the message?

regards
Vivek

>
> Robin.
>
>
>> + ret = -ENODEV;
>> + goto out_unlink;
>> + }
>> + }
>> +
>> arm_smmu_rpm_put(smmu);
>> return 0;
>> +out_unlink:
>> + iommu_device_unlink(&smmu->iommu, dev);
>> + arm_smmu_master_free_smes(fwspec);
>> out_rpm_put:
>> arm_smmu_rpm_put(smmu);
>> out_cfg_free:
>> @@ -1486,6 +1507,14 @@ static void arm_smmu_remove_device(struct device
>> *dev)
>> cfg = fwspec->iommu_priv;
>> smmu = cfg->smmu;
>> + if (pm_runtime_enabled(smmu->dev)) {
>> + struct device_link *link;
>> +
>> + link = device_link_find(dev, smmu->dev);
>> + if (link)
>> + device_link_del(link);
>> + }
>> +
>> ret = arm_smmu_rpm_get(smmu);
>> if (ret < 0)
>> return;
>>
>

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

2018-03-15 10:45:53

[permalink] [raw]

Subject: Re: [PATCH v9 4/5] iommu/arm-smmu: Add the device_link between masters and smmu

On 15/03/18 06:18, Tomasz Figa wrote:
> On Thu, Mar 15, 2018 at 2:50 AM, Robin Murphy <[email protected]> wrote:
>> On 13/03/18 08:55, Vivek Gautam wrote:
>>>
>>> From: Sricharan R <[email protected]>
>>>
>>> Finally add the device link between the master device and
>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>> master needs it. This is done from add_device callback which gets
>>> called once when the master is added to the smmu.
>>>
>>> Signed-off-by: Sricharan R <[email protected]>
>>> Signed-off-by: Vivek Gautam <[email protected]>
>>> Reviewed-by: Tomasz Figa <[email protected]>
>>> ---
>>> drivers/iommu/arm-smmu.c | 29 +++++++++++++++++++++++++++++
>>> 1 file changed, 29 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>> index 56a04ae80bf3..64953ff2281f 100644
>>> --- a/drivers/iommu/arm-smmu.c
>>> +++ b/drivers/iommu/arm-smmu.c
>>> @@ -1460,10 +1460,31 @@ static int arm_smmu_add_device(struct device *dev)
>>> iommu_device_link(&smmu->iommu, dev);
>>> + if (pm_runtime_enabled(smmu->dev)) {
>>> + struct device_link *link;
>>> +
>>> + /*
>>> + * Establish the link between smmu and master, so that the
>>> + * smmu gets runtime enabled/disabled as per the master's
>>> + * needs.
>>> + */
>>> + link = device_link_add(dev, smmu->dev,
>>> DL_FLAG_PM_RUNTIME);
>>> + if (!link) {
>>
>>
>> FWIW, given that we don't really care about link itself, I'd be quite happy
>> to simplify that lot down to:
>>
>> if (pm_runtime_enabled(smmu_dev) &&
>> !device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME)) {
>>
>>> + dev_warn(smmu->dev,
>>> + "Unable to add link to the consumer
>>> %s\n",
>>> + dev_name(dev));
>>
>>
>> (side note: since device_link_add() already prints a message on success,
>> maybe it could print its own failure message too?)
>
> I think we care whether adding the link succeeded. If it fails to be
> added, we might end up with a complete system lockup on a system with
> power domains.

Well, yeah, that was implicit - the point is that we *only* care about
whether it succeeded or not. Thus we may as well just check for NULL
directly instead of assigning the value as if we were actually going to
do anything with it.

Robin.

2018-03-15 11:13:24

[permalink] [raw]

Subject: Re: [PATCH v9 4/5] iommu/arm-smmu: Add the device_link between masters and smmu

On 15/03/18 08:57, Vivek Gautam wrote:
> Hi Robin,
>
>
> On Wed, Mar 14, 2018 at 11:20 PM, Robin Murphy <[email protected]> wrote:
>> On 13/03/18 08:55, Vivek Gautam wrote:
>>>
>>> From: Sricharan R <[email protected]>
>>>
>>> Finally add the device link between the master device and
>>> smmu, so that the smmu gets runtime enabled/disabled only when the
>>> master needs it. This is done from add_device callback which gets
>>> called once when the master is added to the smmu.
>>>
>>> Signed-off-by: Sricharan R <[email protected]>
>>> Signed-off-by: Vivek Gautam <[email protected]>
>>> Reviewed-by: Tomasz Figa <[email protected]>
>>> ---
>>> drivers/iommu/arm-smmu.c | 29 +++++++++++++++++++++++++++++
>>> 1 file changed, 29 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>>> index 56a04ae80bf3..64953ff2281f 100644
>>> --- a/drivers/iommu/arm-smmu.c
>>> +++ b/drivers/iommu/arm-smmu.c
>>> @@ -1460,10 +1460,31 @@ static int arm_smmu_add_device(struct device *dev)
>>> iommu_device_link(&smmu->iommu, dev);
>>> + if (pm_runtime_enabled(smmu->dev)) {
>>> + struct device_link *link;
>>> +
>>> + /*
>>> + * Establish the link between smmu and master, so that the
>>> + * smmu gets runtime enabled/disabled as per the master's
>>> + * needs.
>>> + */
>>> + link = device_link_add(dev, smmu->dev,
>>> DL_FLAG_PM_RUNTIME);
>>> + if (!link) {
>>
>>
>> FWIW, given that we don't really care about link itself, I'd be quite happy
>> to simplify that lot down to:
>>
>> if (pm_runtime_enabled(smmu_dev) &&
>> !device_link_add(dev, smmu->dev, DL_FLAG_PM_RUNTIME)) {
>
> Sure, will update this.
>
>>
>>> + dev_warn(smmu->dev,
>>> + "Unable to add link to the consumer
>>> %s\n",
>>> + dev_name(dev));
>>
>>
>> (side note: since device_link_add() already prints a message on success,
>> maybe it could print its own failure message too?)
>
> Should we make device_link that verbose - to print failure messages at
> each step (there are atleast a couple where we return link as NULL),
> or we can let the users handle printing the message?

I didn't mean to imply anything more than the idea below (although the
whole function could of course be refactored further to report explicit
error values). It just seems a bit unbalanced for the core code to be
noisy about success yet silent about failure, but ultimately that's an
entirely separate issue which doesn't have to have any bearing on this
series.

Robin.

----->8-----
diff --git a/drivers/base/core.c b/drivers/base/core.c
index b2261f92f2f1..895da95f5cb9 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -275,6 +275,9 @@ struct device_link *device_link_add(struct device
*consumer,
out:
device_pm_unlock();
device_links_write_unlock();
+ if (!link)
+ dev_warn(consumer, "Failed to link to supplier %s\n",
dev_name(supplier));
+
return link;
}
EXPORT_SYMBOL_GPL(device_link_add);

2018-03-20 07:58:25

[permalink] [raw]

Subject: Re: [PATCH v9 1/5] driver core: Find an existing link between two devices

Hi Lukasz,

On 3/14/2018 5:57 PM, Lukas Wunner wrote:
> On Wed, Mar 14, 2018 at 12:14:15PM +0000, Robin Murphy wrote:
>>>> On Wed, Mar 14, 2018 at 8:12 PM, Rafael J. Wysocki <[email protected]> wrote:
>>>>> On Tuesday, March 13, 2018 12:23:34 PM CET Tomasz Figa wrote:
>>>>>> On Tue, Mar 13, 2018 at 7:34 PM, Vivek Gautam <[email protected]> wrote:
>>>>>>> On Tue, Mar 13, 2018 at 3:45 PM, Tomasz Figa <[email protected]> wrote:
>>>>>>>> On Tue, Mar 13, 2018 at 5:55 PM, Vivek Gautam <[email protected]> wrote:
>>>>>>>>> The lists managing the device-links can be traversed to
>>>>>>>>> find the link between two devices. The device_link_add() APIs
>>>>>>>>> does traverse these lists to check if there's already a link
>>>>>>>>> setup between the two devices.
>>>>>>>>> So, add a new APIs, device_link_find(), to find an existing
>>>>>>>>> device link between two devices - suppliers and consumers.
>>>>>>>> I'm wondering if this API would be useful for anything else that the
>>>>>>>> problem we're trying to solve with deleting links without storing them
>>>>>>>> anywhere. Perhaps a device_link_del_dev(consumer, supplier) would be a
>>>>>>>> better alternative?
>>>>>>> Yea, that sounds simpler i think. Will add this API instead of
>>>>>>> find_link(). Thanks.
>>>>>> Perhaps let's wait for a moment to see if there are other opinions. :)
>>>>>>
>>>>>> Rafael, Lucas, any thoughts?
>>>>> It is not clear to me what the device_link_del_dev(consumer, supplier)
>>>>> would do.
>> Not quite - the issue here is that we have one supplier with an arbitrarily
>> large number of consumers, and would prefer that supplier not to have to
>> spend a whole bunch of memory to store all the struct device_link pointers
>> for the sole reason of having something to give to device_link_del() at the
>> end, given that the device links code is already keeping track of everything
>> internally anyway.
> Makes sense to me. How about an additional flag which autoremoves the
> link on provider unbind?

If I understand this correctly, if we create the device link with
DL_FLAG_AUTOREMOVE, the link is deleted after a consumer unbind. During
a supplier unbind all we get is a WARN_ON with DL_FLAG_AUTOREMOVE. I
guess that's an intended behavior?

If this is the case, then the consumer/supplier drivers just don't have
to take care of deleting the device link explicitly.
Is my understanding correct?

regards
Vivek

>
> Thanks,
>
> Lukas

2018-03-20 09:51:36

[permalink] [raw]

Subject: Re: [PATCH v9 3/5] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

Hi Robin,

On 3/14/2018 11:16 PM, Robin Murphy wrote:
> On 13/03/18 08:55, Vivek Gautam wrote:
>> From: Sricharan R <[email protected]>
>>
>> The smmu device probe/remove and add/remove master device callbacks
>> gets called when the smmu is not linked to its master, that is without
>> the context of the master device. So calling runtime apis in those
>> places
>> separately.
>>
>> Signed-off-by: Sricharan R <[email protected]>
>> [vivek: Cleanup pm runtime calls]
>> Signed-off-by: Vivek Gautam <[email protected]>
>> Reviewed-by: Tomasz Figa <[email protected]>
>> ---
>> drivers/iommu/arm-smmu.c | 95
>> ++++++++++++++++++++++++++++++++++++++++++++----
>> 1 file changed, 87 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index d5873d545024..56a04ae80bf3 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -268,6 +268,20 @@ static struct arm_smmu_option_prop
>> arm_smmu_options[] = {
>>       { 0, NULL},
>> };
>> +static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
>> +{
>> +    if (pm_runtime_enabled(smmu->dev))
>> +        return pm_runtime_get_sync(smmu->dev);
>> +
>> +    return 0;
>> +}
>> +
>> +static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
>> +{
>> +    if (pm_runtime_enabled(smmu->dev))
>> +        pm_runtime_put(smmu->dev);
>> +}
>> +
>> static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain
>> *dom)
>> {
>>       return container_of(dom, struct arm_smmu_domain, domain);
>> @@ -913,11 +927,15 @@ static void
>> arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>>       struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>>       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>> -    int irq;
>> +    int ret, irq;
>>       if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
>>           return;
>> +    ret = arm_smmu_rpm_get(smmu);
>> +    if (ret < 0)
>> +        return;
>> +
>>       /*
>>        * Disable the context bank and free the page tables before
>> freeing
>>        * it.
>> @@ -932,6 +950,8 @@ static void
>> arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>>       free_io_pgtable_ops(smmu_domain->pgtbl_ops);
>>       __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
>> +
>> +    arm_smmu_rpm_put(smmu);
>> }
>> static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
>> @@ -1213,10 +1233,15 @@ static int arm_smmu_attach_dev(struct
>> iommu_domain *domain, struct device *dev)
>>           return -ENODEV;
>>       smmu = fwspec_smmu(fwspec);
>> +
>> +    ret = arm_smmu_rpm_get(smmu);
>> +    if (ret < 0)
>> +        return ret;
>> +
>>       /* Ensure that the domain is finalised */
>>       ret = arm_smmu_init_domain_context(domain, smmu);
>>       if (ret < 0)
>> -        return ret;
>> +        goto rpm_put;
>>       /*
>>        * Sanity check the domain. We don't support domains across
>> @@ -1230,29 +1255,47 @@ static int arm_smmu_attach_dev(struct
>> iommu_domain *domain, struct device *dev)
>>       }
>>       /* Looks ok, so add the device to the domain */
>> -    return arm_smmu_domain_add_master(smmu_domain, fwspec);
>> +    ret = arm_smmu_domain_add_master(smmu_domain, fwspec);
>> +
>> +rpm_put:
>> +    arm_smmu_rpm_put(smmu);
>> +    return ret;
>> }
>> static int arm_smmu_map(struct iommu_domain *domain, unsigned
>> long iova,
>>               phys_addr_t paddr, size_t size, int prot)
>> {
>>       struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
>> +    struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>> +    struct arm_smmu_device *smmu = smmu_domain->smmu;
>
> Nit: please use arm_smmu_domain for ops as well (as it was before
> 523d7423e21b), or consistently elide it for smmu - the mixture of both
> methods is just a horrible mess (here and in unmap).

Sure, will make it consistent for arm_smmu_device (in both places -
map/unmap)

     struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu;

>
>> +    int ret;
>>       if (!ops)
>>           return -ENODEV;
>> -    return ops->map(ops, iova, paddr, size, prot);
>> +    arm_smmu_rpm_get(smmu);
>> +    ret = ops->map(ops, iova, paddr, size, prot);
>> +    arm_smmu_rpm_put(smmu);
>> +
>> +    return ret;
>> }
>> static size_t arm_smmu_unmap(struct iommu_domain *domain,
>> unsigned long iova,
>>                    size_t size)
>> {
>>       struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
>> +    struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>> +    struct arm_smmu_device *smmu = smmu_domain->smmu;
>> +    size_t ret;
>>       if (!ops)
>>           return 0;
>> -    return ops->unmap(ops, iova, size);
>> +    arm_smmu_rpm_get(smmu);
>> +    ret = ops->unmap(ops, iova, size);
>> +    arm_smmu_rpm_put(smmu);
>> +
>> +    return ret;
>> }
>> static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
>> @@ -1407,14 +1450,22 @@ static int arm_smmu_add_device(struct device
>> *dev)
>>       while (i--)
>>           cfg->smendx[i] = INVALID_SMENDX;
>> +    ret = arm_smmu_rpm_get(smmu);
>> +    if (ret < 0)
>> +        goto out_cfg_free;
>> +
>>       ret = arm_smmu_master_alloc_smes(dev);
>
> Nit: it would be easier to just do the rpm_put here; then you don't
> need to mess with the cleanup path.

Sure, will do that. It will be cleaner.

>
>>       if (ret)
>> -        goto out_cfg_free;
>> +        goto out_rpm_put;
>>       iommu_device_link(&smmu->iommu, dev);
>> +    arm_smmu_rpm_put(smmu);
>> +
>>       return 0;
>> +out_rpm_put:
>> +    arm_smmu_rpm_put(smmu);
>> out_cfg_free:
>>       kfree(cfg);
>> out_free:
>> @@ -1427,7 +1478,7 @@ static void arm_smmu_remove_device(struct
>> device *dev)
>>       struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>>       struct arm_smmu_master_cfg *cfg;
>>       struct arm_smmu_device *smmu;
>> -
>> +    int ret;
>>       if (!fwspec || fwspec->ops != &arm_smmu_ops)
>>           return;
>> @@ -1435,8 +1486,15 @@ static void arm_smmu_remove_device(struct
>> device *dev)
>>       cfg = fwspec->iommu_priv;
>>       smmu = cfg->smmu;
>> +    ret = arm_smmu_rpm_get(smmu);
>> +    if (ret < 0)
>> +        return;
>> +
>>       iommu_device_unlink(&smmu->iommu, dev);
>>       arm_smmu_master_free_smes(fwspec);
>> +
>> +    arm_smmu_rpm_put(smmu);
>> +
>>       iommu_group_remove_device(dev);
>>       kfree(fwspec->iommu_priv);
>>       iommu_fwspec_free(dev);
>> @@ -2124,6 +2182,8 @@ static int arm_smmu_device_probe(struct
>> platform_device *pdev)
>>           smmu->irqs[i] = irq;
>>       }
>> +    platform_set_drvdata(pdev, smmu);
>> +
>>       err = devm_clk_bulk_get(smmu->dev, smmu->num_clks, smmu->clks);
>>       if (err)
>>           return err;
>> @@ -2132,6 +2192,19 @@ static int arm_smmu_device_probe(struct
>> platform_device *pdev)
>>       if (err)
>>           return err;
>> +    /*
>> +     * We want to avoid touching dev->power.lock in fastpaths unless
>> +     * it's really going to do something useful - pm_runtime_enabled()
>> +     * can serve as an ideal proxy for that decision. So, conditionally
>> +     * enable pm_runtime.
>> +     */
>> +    if (dev->pm_domain)
>> +        pm_runtime_enable(dev);
>> +
>> +    err = arm_smmu_rpm_get(smmu);
>> +    if (err < 0)
>> +        return err;
>> +
>>       err = arm_smmu_device_cfg_probe(smmu);
>>       if (err)
>>           return err;
>> @@ -2173,10 +2246,11 @@ static int arm_smmu_device_probe(struct
>> platform_device *pdev)
>>           return err;
>>       }
>> -    platform_set_drvdata(pdev, smmu);
>>       arm_smmu_device_reset(smmu);
>>       arm_smmu_test_smr_masks(smmu);
>> +    arm_smmu_rpm_put(smmu);
>> +
>>       /*
>>        * For ACPI and generic DT bindings, an SMMU will be probed before
>>        * any device which might need it, so we want the bus ops in place
>> @@ -2212,8 +2286,13 @@ static int arm_smmu_device_remove(struct
>> platform_device *pdev)
>>       if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
>>           dev_err(&pdev->dev, "removing device with active domains!\n");
>> +    arm_smmu_rpm_get(smmu);
>>       /* Turn the thing off */
>>       writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>> +    arm_smmu_rpm_put(smmu);
>> +
>> +    if (pm_runtime_enabled(smmu->dev))
>> +        pm_runtime_disable(smmu->dev);
>>       clk_bulk_unprepare(smmu->num_clks, smmu->clks);
>
> I don't know how runtime and system PM interact - does the reset in
> arm_smmu_pm_resume need special treatment as well, or is the device
> guaranteed to be powered up at that point by other means?

So, as Tomasz wrote, we should be okay with this.
Thanks.

regards
Vivek
>
> Robin.
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-arm-msm" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2018-03-28 06:13:24

by Yisheng Xie

[permalink] [raw]

Subject: Re: [PATCH v9 5/5] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

Hi Vivek,

On 2018/3/28 12:37, Vivek Gautam wrote:
> Hi Yisheng
>
>
> On 3/28/2018 6:54 AM, Yisheng Xie wrote:
>> Hi Vivek,
>>
>> On 2018/3/13 16:55, Vivek Gautam wrote:
>>> +- power-domains: Specifiers for power domains required to be powered on for
>>> + the SMMU to operate, as per generic power domain bindings.
>>> +
>> In this patchset, power-domains is not used right? And you just do the clock gating,
>> but not power gating, right?
>
> We are handling the power-domains too. Please see the example in this binding doc.

I see, but I do not find the point in code of these patchset, do you mean PMIC(e.g mmcc)
will gate the power domain of SMMU(e.g. MDSS_GDSC of mmcc) when PMIC suspend?

>
>>
>> Another question is if smmu do power gating, it will reset some of its registers, so
>> it need save at suspend and restore at resume, right?
>
> Qualcomm implementation of the arm-smmu has the retenetion enabled. So the smmu doesn't
> loose state when power is pulled out of it.
> And now we are just selectively enabling the runtime pm. So only the platforms that can really
> support runtime pm can enable it.

Get it, thanks for your explain.

Thanks
Yisheng
>
> Thanks
> Vivek
>>
>> Thanks
>> Yisheng
>>
>
>
>

2018-04-10 13:19:50

[permalink] [raw]

Subject: Re: [PATCH v9 5/5] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

Hi Yisheng,

Sorry, I think we missed your question here.

On Wed, Mar 28, 2018 at 3:12 PM Yisheng Xie <[email protected]> wrote:

> Hi Vivek,

> On 2018/3/28 12:37, Vivek Gautam wrote:
> > Hi Yisheng
> >
> >
> > On 3/28/2018 6:54 AM, Yisheng Xie wrote:
> >> Hi Vivek,
> >>
> >> On 2018/3/13 16:55, Vivek Gautam wrote:
> >>> +- power-domains: Specifiers for power domains required to be
powered on for
> >>> + the SMMU to operate, as per generic power domain
bindings.
> >>> +
> >> In this patchset, power-domains is not used right? And you just do the
clock gating,
> >> but not power gating, right?
> >
> > We are handling the power-domains too. Please see the example in this
binding doc.

> I see, but I do not find the point in code of these patchset, do you mean
PMIC(e.g mmcc)
> will gate the power domain of SMMU(e.g. MDSS_GDSC of mmcc) when PMIC
suspend?

If respective SoC power domains is registered as a standard genpd PM
domain, then the runtime PM subsystem will take care of power domain
control at runtime suspend and resume.

Best regards,
Tomasz

2018-04-11 01:27:41

by Yisheng Xie

[permalink] [raw]

Subject: Re: [PATCH v9 5/5] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

Hi Tomasz,

On 2018/4/10 21:14, Tomasz Figa wrote:
> Hi Yisheng,
>
> Sorry, I think we missed your question here.
>
> On Wed, Mar 28, 2018 at 3:12 PM Yisheng Xie <[email protected]> wrote:
>
>> Hi Vivek,
>
>> On 2018/3/28 12:37, Vivek Gautam wrote:
>>> Hi Yisheng
>>>
>>>
>>> On 3/28/2018 6:54 AM, Yisheng Xie wrote:
>>>> Hi Vivek,
>>>>
>>>> On 2018/3/13 16:55, Vivek Gautam wrote:
>>>>> +- power-domains: Specifiers for power domains required to be
> powered on for
>>>>> + the SMMU to operate, as per generic power domain
> bindings.
>>>>> +
>>>> In this patchset, power-domains is not used right? And you just do the
> clock gating,
>>>> but not power gating, right?
>>>
>>> We are handling the power-domains too. Please see the example in this
> binding doc.
>
>> I see, but I do not find the point in code of these patchset, do you mean
> PMIC(e.g mmcc)
>> will gate the power domain of SMMU(e.g. MDSS_GDSC of mmcc) when PMIC
> suspend?
>
>
> If respective SoC power domains is registered as a standard genpd PM
> domain, then the runtime PM subsystem will take care of power domain
> control at runtime suspend and resume.
>

Get it, thanks for your explain, I should have learned about this.

Thanks
Yisheng

> Best regards,
> Tomasz
>
> .
>

2018-04-11 05:18:48