Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Thu, 5 May 2022 14:38:39 +0100
From:   Jean-Philippe Brucker <jean-philippe@linaro.org>
To:     Baolu Lu <baolu.lu@linux.intel.com>
Cc:     Joerg Roedel <joro@8bytes.org>, Jason Gunthorpe <jgg@nvidia.com>,
        Christoph Hellwig <hch@infradead.org>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v5 10/12] iommu: Prepare IOMMU domain for IOPF
Message-ID: <YnPTXwjpHnnGp4j2@myrica>
References: <20220502014842.991097-1-baolu.lu@linux.intel.com>
 <20220502014842.991097-11-baolu.lu@linux.intel.com>
 <YnFydE8j8l7Q4m+b@myrica>
 <9144a782-04d2-a09d-4ac1-7133e5986619@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <9144a782-04d2-a09d-4ac1-7133e5986619@linux.intel.com>
Precedence: bulk

Hi Baolu,

On Thu, May 05, 2022 at 04:31:38PM +0800, Baolu Lu wrote:
> On 2022/5/4 02:20, Jean-Philippe Brucker wrote:
> > > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> > > index 7cae631c1baa..33449523afbe 100644
> > > --- a/drivers/iommu/iommu.c
> > > +++ b/drivers/iommu/iommu.c
> > > @@ -3174,3 +3174,24 @@ void iommu_detach_device_pasid(struct iommu_domain *domain,
> > >   	iommu_group_put(group);
> > >   }
> > > +
> > > +struct iommu_domain *iommu_get_domain_for_dev_pasid(struct device *dev,
> > > +						    ioasid_t pasid)
> > > +{
> > > +	struct iommu_domain *domain;
> > > +	struct iommu_group *group;
> > > +
> > > +	if (!pasid_valid(pasid))
> > > +		return NULL;
> > > +
> > > +	group = iommu_group_get(dev);
> > > +	if (!group)
> > > +		return NULL;
> > > +
> > > +	mutex_lock(&group->mutex);
> > Unfortunately this still causes the deadlock when unbind() flushes the
> > IOPF queue while holding the group mutex.
> 
> Sorry, I didn't get your point here.
> 
> Do you mean unbind() could hold group mutex before calling this helper?
> The group mutex is only available in iommu.c. The unbind() has no means
> to hold this lock. Or, I missed anything?

I wasn't clear, it's iommu_detach_device_pasid() that holds the
group->mutex:

 iommu_sva_unbind_device()          |
  iommu_detach_device_pasid()       |
   mutex_lock(&group->mutex)        |
   domain->ops->detach_dev_pasid()  | iopf_handle_group()
    iopf_queue_flush_dev()          |  iommu_get_domain_for_dev_pasid()
     ... wait for IOPF work         |   mutex_lock(&group->mutex)
                                    |    ... deadlock

Thanks,
Jean

> 
> Best regards,
> baolu
> 
> > 
> > If we make this function private to IOPF, then we can get rid of this
> > mutex_lock(). It's OK because:
> > 
> > * xarray protects its internal state with RCU, so we can call
> >    xa_load() outside the lock.
> > 
> > * The domain obtained from xa_load is finalized. Its content is valid
> >    because xarray stores the domain using rcu_assign_pointer(), which has a
> >    release memory barrier, which pairs with data dependencies in IOPF
> >    (domain->sva_ioas etc).
> > 
> >    We'll need to be careful about this when allowing other users to install
> >    a fault handler. Should be fine as long as the handler and data are
> >    installed before the domain is added to pasid_array.
> > 
> > * We know the domain is valid the whole time IOPF is using it, because
> >    unbind() waits for pending faults.
> > 
> > We just need a comment explaining the last point, something like:
> > 
> >         /*
> > 	* Safe to fetch outside the group mutex because:
> >          * - xarray protects its internal state with RCU
> >          * - the domain obtained is either NULL or fully formed
> > 	* - the IOPF work is the only caller and is flushed before the
> > 	*   domain is freed.
> >          */
> > 
> > Thanks,
> > Jean
> > 
> > > +	domain = xa_load(&group->pasid_array, pasid);
> > > +	mutex_unlock(&group->mutex);
> > > +	iommu_group_put(group);
> > > +
> > > +	return domain;
> > > +}
>