Received: by 10.223.176.46 with SMTP id f43csp432648wra; Thu, 18 Jan 2018 20:02:45 -0800 (PST) X-Google-Smtp-Source: ACJfBouNA2kDUrSYUHRQH3aJwYwEhBVsKLfjZCgOTqAG+0OqjKfCkaVfp3DpUWC7Ct7EFsoPPiR/ X-Received: by 10.98.42.23 with SMTP id q23mr44883697pfq.161.1516334565547; Thu, 18 Jan 2018 20:02:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516334565; cv=none; d=google.com; s=arc-20160816; b=qPLyPMFRuGkyqL1T9Muaau9qPtmYyRePqtbj4P40mKfjdW5w3gWZLx3vfB3lzlFo5Q 6/f1i0WK01wNZEyozktAt/VOdaGVz0PiIvVH7Dk59+6tdvBaFw5g8iCv+arn9mei6G95 AoZD88WGcY5+jr+tkMHldTE97qU5+rMuVs1mfx1mDBXDMeNCXZ4U1M6U6G6+4ZdXHVyA hk0AXHSXrpb0ByduAvzLChXKBx0jmAl2ZRPyYN0RIoZ3H2Nxx8JNOrFXHoPiSaVuEADi N6K0lU+BgbWDPjkYcQVIH85dLVay+xKNj1QKkTIYs2fUEKbg+8L4tuGF24/lTMXngwMz LMjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:dkim-signature :arc-authentication-results; bh=Bq0qJ4zWzmkQc8ji6OcQVTagoGhwC9whlyTH0Nlhc88=; b=hbHoadHq4U0bkNcSlKDsWmXa/MFN4vfTGBKfhfMSU/0ONleQZ/jBYaDplzvdtBgwJq nwp+lpxPAYQEm7Uo/gvaH12p3YleaR45WPvMLARt9Pu9TrUYRhGOJX2GcgpQpn/wVAMs Vkp5XrxN6QQPYQGb+ag5IiOOOxvbFkt9on58R1Mqw6zfpT1dXkd4P+8E5dKXyTaxIGTQ thXK2CJP2F7+g/lF9rWOBYgnA+IwMmlT9gtsb05wqyhSGCrUy1C80wbKyxKJl4iB7jPQ DC1Kcs0BqZriSQsvoGNxmb9EiT82bo7ZIsTW28Py4fnKHtTodqoKXny6EJTayVfnidC8 NbhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=PEWxpc0Y; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z1si7477147pgs.409.2018.01.18.20.02.30; Thu, 18 Jan 2018 20:02:45 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=PEWxpc0Y; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755501AbeASECC (ORCPT + 99 others); Thu, 18 Jan 2018 23:02:02 -0500 Received: from mail-ua0-f195.google.com ([209.85.217.195]:41022 "EHLO mail-ua0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754836AbeASEB5 (ORCPT ); Thu, 18 Jan 2018 23:01:57 -0500 Received: by mail-ua0-f195.google.com with SMTP id x10so264179ual.8 for ; Thu, 18 Jan 2018 20:01:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Bq0qJ4zWzmkQc8ji6OcQVTagoGhwC9whlyTH0Nlhc88=; b=PEWxpc0YdiPvSLQATFezEA2ImN3AixaQ10kJ1exEHQKSpCJm8boHD9TekmQMpNhR6n IJ4WsQlb55qBFOvQtN5WU4JtPkqt2nu8GIAH/4FmZQa+jaj7lCIF007bNT8/Y9/R6Epz Hf8xru1sk2mGR5KcAN22sPGjH1c87CSSx2jxA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Bq0qJ4zWzmkQc8ji6OcQVTagoGhwC9whlyTH0Nlhc88=; b=Pcdvx8LZgiWCPWWYu/dA9H+EaPmYiFDS1O5fWZmFS94H/gRGg5yZ7sbjr/na9Ajm6n n8G1Ys5WzHnar7IK/hKVf4QQNYk2x4Cby/VHy+7AYCcEDNpnmZ7Xtct2ZEJnFB2DQRBp wP9e+gY09+wtjlV2RkE14xWrL888+ji7mkg2mMAEbZyP7Pqm7wtBwokgDuSb2WSpZf7b zvmXasPgtkFAja/PhyTknxzFQoZVNFFAPQo39VjfQdclMQ6AyDBzhreIKKYT4XQ9/YwV r0v4HXFz+HoYjA0mtpIym5avh5fq6m2S9VOf5CYjdYbOMD/ij5NaKt0jZf1py5yXCEFu Fsjg== X-Gm-Message-State: AKwxyteic1iEazq7ycCuTrmpRUx1MwONYhUAm/yHMHas16AK388LbJrG 86npPIS2rn2oFW+8+dsm8cwjO0qToC8= X-Received: by 10.176.89.37 with SMTP id n34mr6494184uad.73.1516334516010; Thu, 18 Jan 2018 20:01:56 -0800 (PST) Received: from mail-ua0-f182.google.com (mail-ua0-f182.google.com. [209.85.217.182]) by smtp.gmail.com with ESMTPSA id m2sm3538569uad.16.2018.01.18.20.01.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Jan 2018 20:01:54 -0800 (PST) Received: by mail-ua0-f182.google.com with SMTP id t6so265226ual.7 for ; Thu, 18 Jan 2018 20:01:54 -0800 (PST) X-Received: by 10.176.85.8 with SMTP id t8mr7116453uaa.49.1516334513663; Thu, 18 Jan 2018 20:01:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.159.45.142 with HTTP; Thu, 18 Jan 2018 20:01:33 -0800 (PST) In-Reply-To: <20180118115251.5542-13-jeffy.chen@rock-chips.com> References: <20180118115251.5542-1-jeffy.chen@rock-chips.com> <20180118115251.5542-13-jeffy.chen@rock-chips.com> From: Tomasz Figa Date: Fri, 19 Jan 2018 13:01:33 +0900 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v4 12/13] iommu/rockchip: Add runtime PM support To: Jeffy Chen Cc: linux-kernel@vger.kernel.org, Ricky Liang , Robin Murphy , simon xue , Heiko Stuebner , "open list:ARM/Rockchip SoC..." , "open list:IOMMU DRIVERS" , Joerg Roedel , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jan 18, 2018 at 8:52 PM, Jeffy Chen wrote: > When the power domain is powered off, the IOMMU cannot be accessed and > register programming must be deferred until the power domain becomes > enabled. > > Add runtime PM support, and use runtime PM device link from IOMMU to > master to startup and shutdown IOMMU. > > Signed-off-by: Jeffy Chen > --- > > Changes in v4: None > Changes in v3: > Only call startup() and shutdown() when iommu attached. > Remove pm_mutex. > Check runtime PM disabled. > Check pm_runtime in rk_iommu_irq(). > > Changes in v2: None > > drivers/iommu/rockchip-iommu.c | 180 ++++++++++++++++++++++++++++++++--------- > 1 file changed, 141 insertions(+), 39 deletions(-) > > diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c > index 2c095f96c033..e2e7acc3039d 100644 > --- a/drivers/iommu/rockchip-iommu.c > +++ b/drivers/iommu/rockchip-iommu.c > @@ -22,6 +22,7 @@ > #include > #include > #include > +#include > #include > #include > > @@ -99,6 +100,7 @@ struct rk_iommu { > }; > > struct rk_iommudata { > + struct device_link *link; /* runtime PM link from IOMMU to master */ > struct rk_iommu *iommu; > }; > > @@ -583,7 +585,11 @@ static irqreturn_t rk_iommu_irq(int irq, void *dev_id) > u32 int_status; > dma_addr_t iova; > irqreturn_t ret = IRQ_NONE; > - int i; > + int i, err; > + > + err = pm_runtime_get_if_in_use(iommu->dev); > + if (err <= 0 && err != -EINVAL) > + return ret; > > WARN_ON(rk_iommu_enable_clocks(iommu)); > > @@ -635,6 +641,9 @@ static irqreturn_t rk_iommu_irq(int irq, void *dev_id) > > rk_iommu_disable_clocks(iommu); > > + if (pm_runtime_enabled(iommu->dev)) > + pm_runtime_put(iommu->dev); I think this might be racy. There are some places where pm_runtime_enable/disable() are called on devices implicitly and I'm not sure if we're guaranteed that they don't happen between our pm_runtime_get_if_in_use() and pm_runtime_enabled() calls. An example of a race-free solution would be to save the pm_runtime_get_if_in_use() result to a local variable (e.g. bool need_runtime_put) and then call pm_runtime_put() based on that. > + > return ret; > } > > @@ -676,10 +685,20 @@ static void rk_iommu_zap_iova(struct rk_iommu_domain *rk_domain, > spin_lock_irqsave(&rk_domain->iommus_lock, flags); > list_for_each(pos, &rk_domain->iommus) { > struct rk_iommu *iommu; > + int ret; > + > iommu = list_entry(pos, struct rk_iommu, node); > - rk_iommu_enable_clocks(iommu); > - rk_iommu_zap_lines(iommu, iova, size); > - rk_iommu_disable_clocks(iommu); > + > + /* Only zap TLBs of IOMMUs that are powered on. */ > + ret = pm_runtime_get_if_in_use(iommu->dev); > + if (ret > 0 || ret == -EINVAL) { > + rk_iommu_enable_clocks(iommu); > + rk_iommu_zap_lines(iommu, iova, size); > + rk_iommu_disable_clocks(iommu); > + } > + > + if (ret > 0) > + pm_runtime_put(iommu->dev); This one nicely avoids the race I mentioned above. :) > } > spin_unlock_irqrestore(&rk_domain->iommus_lock, flags); > } > @@ -882,22 +901,30 @@ static struct rk_iommu *rk_iommu_from_dev(struct device *dev) > return data ? data->iommu : NULL; > } > [snip] > + spin_lock_irqsave(&rk_domain->iommus_lock, flags); > + list_add_tail(&iommu->node, &rk_domain->iommus); > + spin_unlock_irqrestore(&rk_domain->iommus_lock, flags); > > - dev_dbg(dev, "Detached from iommu domain\n"); > + ret = pm_runtime_get_if_in_use(iommu->dev); > + if (ret <= 0 && ret != -EINVAL) > + return 0; > + > + ret = rk_iommu_startup(iommu); > + if (ret) > + rk_iommu_detach_device(iommu->domain, dev); > + > + if (pm_runtime_enabled(iommu->dev)) > + pm_runtime_put(iommu->dev); Here we should also probably act based on what pm_runtime_get_if_in_use() returned rather than asking pm_runtime_enabled(). Best regards, Tomasz