Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <9144a782-04d2-a09d-4ac1-7133e5986619@linux.intel.com>
Date:   Thu, 5 May 2022 16:31:38 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.1
Subject: Re: [PATCH v5 10/12] iommu: Prepare IOMMU domain for IOPF
Content-Language: en-US
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     Joerg Roedel <joro@8bytes.org>, Jason Gunthorpe <jgg@nvidia.com>,
        Christoph Hellwig <hch@infradead.org>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
References: <20220502014842.991097-1-baolu.lu@linux.intel.com>
 <20220502014842.991097-11-baolu.lu@linux.intel.com> <YnFydE8j8l7Q4m+b@myrica>
From:   Baolu Lu <baolu.lu@linux.intel.com>
In-Reply-To: <YnFydE8j8l7Q4m+b@myrica>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2022/5/4 02:20, Jean-Philippe Brucker wrote:
>> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
>> index 7cae631c1baa..33449523afbe 100644
>> --- a/drivers/iommu/iommu.c
>> +++ b/drivers/iommu/iommu.c
>> @@ -3174,3 +3174,24 @@ void iommu_detach_device_pasid(struct iommu_domain *domain,
>>   
>>   	iommu_group_put(group);
>>   }
>> +
>> +struct iommu_domain *iommu_get_domain_for_dev_pasid(struct device *dev,
>> +						    ioasid_t pasid)
>> +{
>> +	struct iommu_domain *domain;
>> +	struct iommu_group *group;
>> +
>> +	if (!pasid_valid(pasid))
>> +		return NULL;
>> +
>> +	group = iommu_group_get(dev);
>> +	if (!group)
>> +		return NULL;
>> +
>> +	mutex_lock(&group->mutex);
> Unfortunately this still causes the deadlock when unbind() flushes the
> IOPF queue while holding the group mutex.

Sorry, I didn't get your point here.

Do you mean unbind() could hold group mutex before calling this helper?
The group mutex is only available in iommu.c. The unbind() has no means
to hold this lock. Or, I missed anything?

Best regards,
baolu

> 
> If we make this function private to IOPF, then we can get rid of this
> mutex_lock(). It's OK because:
> 
> * xarray protects its internal state with RCU, so we can call
>    xa_load() outside the lock.
> 
> * The domain obtained from xa_load is finalized. Its content is valid
>    because xarray stores the domain using rcu_assign_pointer(), which has a
>    release memory barrier, which pairs with data dependencies in IOPF
>    (domain->sva_ioas etc).
> 
>    We'll need to be careful about this when allowing other users to install
>    a fault handler. Should be fine as long as the handler and data are
>    installed before the domain is added to pasid_array.
> 
> * We know the domain is valid the whole time IOPF is using it, because
>    unbind() waits for pending faults.
> 
> We just need a comment explaining the last point, something like:
> 
>         /*
> 	* Safe to fetch outside the group mutex because:
>          * - xarray protects its internal state with RCU
>          * - the domain obtained is either NULL or fully formed
> 	* - the IOPF work is the only caller and is flushed before the
> 	*   domain is freed.
>          */
> 
> Thanks,
> Jean
> 
>> +	domain = xa_load(&group->pasid_array, pasid);
>> +	mutex_unlock(&group->mutex);
>> +	iommu_group_put(group);
>> +
>> +	return domain;
>> +}