Received-SPF: pass (google.com: domain of linux-kernel+bounces-140040-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <b24f3380-1ac3-41c4-9163-56c1dcff6297@linux.intel.com>
Date: Thu, 11 Apr 2024 15:55:50 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Cc: baolu.lu@linux.intel.com, Joerg Roedel <joro@8bytes.org>,
 Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
 Kevin Tian <kevin.tian@intel.com>, Tina Zhang <tina.zhang@intel.com>,
 Yi Liu <yi.l.liu@intel.com>, iommu@lists.linux.dev,
 linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 12/12] iommu/vt-d: Retire struct intel_svm
To: Jason Gunthorpe <jgg@ziepe.ca>
References: <20240410020844.253535-1-baolu.lu@linux.intel.com>
 <20240410020844.253535-13-baolu.lu@linux.intel.com>
 <20240410154951.GH223006@ziepe.ca>
Content-Language: en-US
From: Baolu Lu <baolu.lu@linux.intel.com>
In-Reply-To: <20240410154951.GH223006@ziepe.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 2024/4/10 23:49, Jason Gunthorpe wrote:
> On Wed, Apr 10, 2024 at 10:08:44AM +0800, Lu Baolu wrote:
>> The struct intel_svm was used for keeping attached devices info for sva
>> domain. Since sva domain is a kind of iommu_domain, the struct
>> dmar_domain should centralize all info of a sva domain, including the
>> info of attached devices. Therefore, retire struct intel_svm to clean up
>> the code.
>>
>> Besides, allocate sva domain in domain_alloc_sva() callback which allows
>> the memory management notifier lifetime to follow the lifetime of the
>> iommu_domain.
>>
>> Co-developed-by: Tina Zhang <tina.zhang@intel.com>
>> Signed-off-by: Tina Zhang <tina.zhang@intel.com>
>> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>> ---
>>   drivers/iommu/intel/iommu.h | 26 ++++------
>>   drivers/iommu/intel/iommu.c |  9 +---
>>   drivers/iommu/intel/svm.c   | 94 +++++++++----------------------------
>>   3 files changed, 32 insertions(+), 97 deletions(-)
> 
> Happy to see the pasid xarray in the driver go away.

Yeah, it was really ugly.

> 
>> @@ -4388,14 +4386,8 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
>>   	WARN_ON_ONCE(!dev_pasid);
>>   	spin_unlock_irqrestore(&dmar_domain->lock, flags);
>>   
>> -	/*
>> -	 * The SVA implementation needs to handle its own stuffs like the mm
>> -	 * notification. Before consolidating that code into iommu core, let
>> -	 * the intel sva code handle it.
>> -	 */
>>   	if (domain->type == IOMMU_DOMAIN_SVA) {
>>   		cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid);
>> -		intel_svm_remove_dev_pasid(domain);
>>   	} else {
>>   		did = domain_id_iommu(dmar_domain, iommu);
>>   		cache_tag_unassign_domain(dmar_domain, did, dev, pasid);
> 
> It seems very strange that SVA has a different DID scheme, why is
> this? PASID and SVA should not be different at this layer.

The VT-d spec recommends that all SVA domains share a single domain ID.
The PASID is unique to each SVA domain, hence the cache tags are unique.
Currently, the Intel IOMMU driver assigns different domain IDs for all
domains except the SVA type.

Sharing a domain ID is not specific to SVA. In general, for devices
under a single IOMMU, domains with unique PASIDs can share the same
domain ID.

In the long term (also on my task list), we will extend the cache tag
code to support sharing domain IDs and remove the domain type check from
the main code. This will also benefit the nesting case, where user
domains nested on the same parent could share a domain ID.

> 
>> @@ -663,7 +596,12 @@ void intel_svm_page_response(struct device *dev, struct iopf_fault *evt,
>>   
>>   static void intel_svm_domain_free(struct iommu_domain *domain)
>>   {
>> -	kfree(to_dmar_domain(domain));
>> +	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
>> +
>> +	if (dmar_domain->notifier.ops)
>> +		mmu_notifier_unregister(&dmar_domain->notifier, domain->mm);
> 
> This should really use mmu_notifier_put() and a delayed kfree, see my
> part 2 ARM series for how that should look.

Learning from

https://lore.kernel.org/linux-iommu/23-v6-228e7adf25eb+4155-smmuv3_newapi_p2_jgg@nvidia.com/

I will make it like below

diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
index 47e475f67046..41374979eecc 100644
--- a/drivers/iommu/intel/svm.c
+++ b/drivers/iommu/intel/svm.c
@@ -178,9 +178,15 @@ static void intel_mm_release(struct mmu_notifier 
*mn, struct mm_struct *mm)

  }

+static void intel_mm_free_notifier(struct mmu_notifier *mn)
+{
+       kfree(container_of(mn, struct dmar_domain, notifier));
+}
+
  static const struct mmu_notifier_ops intel_mmuops = {
         .release = intel_mm_release,
         .arch_invalidate_secondary_tlbs = 
intel_arch_invalidate_secondary_tlbs,
+       .free_notifier = intel_mm_free_notifier,
  };

  static int intel_svm_set_dev_pasid(struct iommu_domain *domain,
@@ -598,10 +604,8 @@ static void intel_svm_domain_free(struct 
iommu_domain *domain)
  {
         struct dmar_domain *dmar_domain = to_dmar_domain(domain);

-       if (dmar_domain->notifier.ops)
-               mmu_notifier_unregister(&dmar_domain->notifier, domain->mm);
-
-       kfree(dmar_domain);
+       /* dmar_domain free is defered to the mmu free_notifier callback. */
+       mmu_notifier_put(&dmar_domain->notifier);
  }

  static const struct iommu_domain_ops intel_svm_domain_ops = {

> 
> Otherwise I think it looks fine

Thank you for review.

Best regards,
baolu