Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <749a7d62-3e6c-ef5c-beaf-6b7add495740@linux.intel.com>
Date:   Wed, 11 May 2022 15:21:31 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.1
Subject: Re: [PATCH v6 08/12] iommu/sva: Use attach/detach_pasid_dev in SVA
 interfaces
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
References: <20220510061738.2761430-1-baolu.lu@linux.intel.com>
 <20220510061738.2761430-9-baolu.lu@linux.intel.com>
 <20220510152330.GG49344@nvidia.com>
From:   Baolu Lu <baolu.lu@linux.intel.com>
In-Reply-To: <20220510152330.GG49344@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2022/5/10 23:23, Jason Gunthorpe wrote:
> On Tue, May 10, 2022 at 02:17:34PM +0800, Lu Baolu wrote:
> 
>> +/**
>> + * iommu_sva_bind_device() - Bind a process address space to a device
>> + * @dev: the device
>> + * @mm: the mm to bind, caller must hold a reference to mm_users
>> + * @drvdata: opaque data pointer to pass to bind callback
>> + *
>> + * Create a bond between device and address space, allowing the device to access
>> + * the mm using the returned PASID. If a bond already exists between @device and
>> + * @mm, it is returned and an additional reference is taken. Caller must call
>> + * iommu_sva_unbind_device() to release each reference.
>> + *
>> + * iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_SVA) must be called first, to
>> + * initialize the required SVA features.
>> + *
>> + * On error, returns an ERR_PTR value.
>> + */
>> +struct iommu_sva *
>> +iommu_sva_bind_device(struct device *dev, struct mm_struct *mm, void *drvdata)
>> +{
>> +	int ret = -EINVAL;
>> +	struct iommu_sva *handle;
>> +	struct iommu_domain *domain;
>> +
>> +	/*
>> +	 * TODO: Remove the drvdata parameter after kernel PASID support is
>> +	 * enabled for the idxd driver.
>> +	 */
>> +	if (drvdata)
>> +		return ERR_PTR(-EOPNOTSUPP);
> 
> Why is this being left behind? Clean up the callers too please.

Okay, let me try to.

> 
>> +	/* Allocate mm->pasid if necessary. */
>> +	ret = iommu_sva_alloc_pasid(mm, 1, (1U << dev->iommu->pasid_bits) - 1);
>> +	if (ret)
>> +		return ERR_PTR(ret);
>> +
>> +	mutex_lock(&iommu_sva_lock);
>> +	/* Search for an existing bond. */
>> +	handle = xa_load(&dev->iommu->sva_bonds, mm->pasid);
>> +	if (handle) {
>> +		refcount_inc(&handle->users);
>> +		goto out_success;
>> +	}
> 
> How can there be an existing bond?
> 
> dev->iommu is per-device
> 
> The device_group_immutable_singleton() insists on a single device
> group
> 
> Basically 'sva_bonds' is the same thing as the group->pasid_array.

Yes, really.

> 
> Assuming we leave room for multi-device groups this logic should just
> be
> 
> 	group = iommu_group_get(dev);
> 	if (!group)
> 		return -ENODEV;
> 
> 	mutex_lock(&group->mutex);
> 	domain = xa_load(&group->pasid_array, mm->pasid);
> 	if (!domain || domain->type != IOMMU_DOMAIN_SVA || domain->mm != mm)
> 		domain = iommu_sva_alloc_domain(dev, mm);
> 
> ?

Agreed. As a helper in iommu core, how about making it more generic like
below?

+struct iommu_domain *iommu_get_domain_for_dev_pasid(struct device *dev,
+                                                   iosid_t pasid,
+                                                   unsigned int type)
+{
+       struct iommu_domain *domain;
+       struct iommu_group *group;
+
+       if (!pasid_valid(pasid))
+               return NULL;
+
+       group = iommu_group_get(dev);
+       if (!group)
+               return NULL;
+
+       mutex_lock(&group->mutex);
+       domain = xa_load(&group->pasid_array, pasid);
+       if (domain && domain->type != type)
+               domain = NULL;
+       mutex_unlock(&group->mutex);
+       iommu_group_put(group);
+
+       return domain;
+}

> 
> And stick the refcount in the sva_domain
> 
> Also, given the current arrangement it might make sense to have a
> struct iommu_domain_sva given that no driver is wrappering this in
> something else.

Fair enough. How about below wrapper?

+struct iommu_sva_domain {
+       /*
+        * Common iommu domain header, *must* be put at the top
+        * of the structure.
+        */
+       struct iommu_domain domain;
+       struct mm_struct *mm;
+       struct iommu_sva bond;
+}

The refcount is wrapped in bond.

Best regards,
baolu