Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org B6B1660712
Date:   Thu, 22 Feb 2018 16:52:00 -0700
From:   Jordan Crouse <jcrouse@codeaurora.org>
To:     Vivek Gautam <vivek.gautam@codeaurora.org>
Cc:     joro@8bytes.org, robh+dt@kernel.org, mark.rutland@arm.com,
        rjw@rjwysocki.net, robin.murphy@arm.com, will.deacon@arm.com,
        robdclark@gmail.com, iommu@lists.linux-foundation.org,
        devicetree@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-pm@vger.kernel.org, dri-devel@lists.freedesktop.org,
        freedreno@lists.freedesktop.org, airlied@linux.ie,
        gregkh@linuxfoundation.org, sboyd@codeaurora.org,
        linux-arm-msm@vger.kernel.org
Subject: Re: [PATCH v7 3/6] iommu/arm-smmu: Invoke pm_runtime during probe,
 add/remove device
Message-ID: <20180222235200.GA18743@jcrouse-lnx.qualcomm.com>
Mail-Followup-To: Vivek Gautam <vivek.gautam@codeaurora.org>,
        joro@8bytes.org, robh+dt@kernel.org, mark.rutland@arm.com,
        rjw@rjwysocki.net, robin.murphy@arm.com, will.deacon@arm.com,
        robdclark@gmail.com, iommu@lists.linux-foundation.org,
        devicetree@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-pm@vger.kernel.org, dri-devel@lists.freedesktop.org,
        freedreno@lists.freedesktop.org, airlied@linux.ie,
        gregkh@linuxfoundation.org, sboyd@codeaurora.org,
        linux-arm-msm@vger.kernel.org
References: <1517999482-17317-1-git-send-email-vivek.gautam@codeaurora.org>
 <1517999482-17317-4-git-send-email-vivek.gautam@codeaurora.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1517999482-17317-4-git-send-email-vivek.gautam@codeaurora.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Feb 07, 2018 at 04:01:19PM +0530, Vivek Gautam wrote:
> From: Sricharan R <sricharan@codeaurora.org>
> 
> The smmu device probe/remove and add/remove master device callbacks
> gets called when the smmu is not linked to its master, that is without
> the context of the master device. So calling runtime apis in those places
> separately.
> 
> Signed-off-by: Sricharan R <sricharan@codeaurora.org>
> [vivek: Cleanup pm runtime calls]
> Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org>
> ---
>  drivers/iommu/arm-smmu.c | 42 ++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 38 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index 9e2f917e16c2..c024f69c1682 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -913,11 +913,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>  	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>  	struct arm_smmu_device *smmu = smmu_domain->smmu;
>  	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> -	int irq;
> +	int ret, irq;
>  
>  	if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
>  		return;
>  
> +	ret = pm_runtime_get_sync(smmu->dev);
> +	if (ret)
> +		return;
> +
>  	/*
>  	 * Disable the context bank and free the page tables before freeing
>  	 * it.
> @@ -932,6 +936,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>  
>  	free_io_pgtable_ops(smmu_domain->pgtbl_ops);
>  	__arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
> +
> +	pm_runtime_put_sync(smmu->dev);
>  }
>  
>  static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> @@ -1407,14 +1413,22 @@ static int arm_smmu_add_device(struct device *dev)
>  	while (i--)
>  		cfg->smendx[i] = INVALID_SMENDX;
>  
> -	ret = arm_smmu_master_alloc_smes(dev);
> +	ret = pm_runtime_get_sync(smmu->dev);
>  	if (ret)
>  		goto out_cfg_free;

Hey Vivek, I just hit a problem with this on sdm845. It turns out that
pm_runtime_get_sync() returns a positive 1 if the device is already active.

I hit this in the GPU code. The a6xx has two platform devices that each use a
different sid on the iommu. The GPU is probed normally from a platform driver
and it in turn initializes the GMU device by way of a phandle.

Because the GMU isn't probed with a platform driver we need to call
of_dma_configure() on the device to set up the IOMMU for the device which ends
up calling through this path and we discover that the smmu->dev is already
powered (pm_runtime_get_sync returns 1).

I'm not immediately sure if this is a bug on sdm845 or not because a cursory
inspection says that the SMMU device shouldn't be powered at this time but there
might be a connection that I'm not seeing. Obviously if the SMMU was left
powered thats a bad thing. But putting that aside it is obvious that this
code should be accommodating of the possibility that the device is already
powered, and so this should be

if (ret < 0)
	goto out_cfg_free;

With that the GPU/GMU successfully comes up on Sean Paul's display testing
branch.

Jordan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project