Received-SPF: pass (google.com: domain of linux-kernel+bounces-133923-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <bc76bb64-8304-4fdf-ae16-03f3e545fd67@linux.intel.com>
Date: Sat, 6 Apr 2024 20:55:44 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
From: Baolu Lu <baolu.lu@linux.intel.com>
Subject: Re: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface
To: "Tian, Kevin" <kevin.tian@intel.com>, Joerg Roedel <joro@8bytes.org>,
 Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
 Jason Gunthorpe <jgg@ziepe.ca>
Cc: baolu.lu@linux.intel.com, "Zhang, Tina" <tina.zhang@intel.com>,
 "Liu, Yi L" <yi.l.liu@intel.com>,
 "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20240325021705.249769-1-baolu.lu@linux.intel.com>
 <20240325021705.249769-2-baolu.lu@linux.intel.com>
 <BN9PR11MB5276A6DDE82623A32FF80F4F8C3B2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Language: en-US
In-Reply-To: <BN9PR11MB5276A6DDE82623A32FF80F4F8C3B2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Kevin,

Thanks for your review comments.

On 3/28/24 3:12 PM, Tian, Kevin wrote:
>> From: Lu Baolu <baolu.lu@linux.intel.com>
>> Sent: Monday, March 25, 2024 10:17 AM
>>
>> +enum cache_tag_type {
>> +	CACHE_TAG_TYPE_IOTLB,
>> +	CACHE_TAG_TYPE_DEVTLB,
>> +	CACHE_TAG_TYPE_PARENT_IOTLB,
>> +	CACHE_TAG_TYPE_PARENT_DEVTLB,
>> +};
> 
> '_TYPE_' can be removed to make it shorter

Okay.

> 
>> +
>> +/* Checks if an existing cache tag can be reused for a new association. */
>> +static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id,
>> +			       struct intel_iommu *iommu, struct device *dev,
>> +			       ioasid_t pasid, enum cache_tag_type type)
> 
> cache_tage_match()

Okay.

> 
>> +{
>> +	if (tag->type != type)
>> +		return false;
>> +
>> +	if (tag->domain_id != domain_id || tag->pasid != pasid)
>> +		return false;
>> +
>> +	if (type == CACHE_TAG_TYPE_IOTLB)
>> +		return tag->iommu == iommu;
>> +
>> +	if (type == CACHE_TAG_TYPE_DEVTLB)
>> +		return tag->dev == dev;
>> +
>> +	return false;
> 
> why do you disallow PARENT_TYPE from reusing? It's not uncommon
> to have two devices attached to a same nested domain hence with
> the same parent domain. Disallowing tag reuse implies unnecessarily
> duplicated cache flushes...

PARENT_TYPE could be reused. The new helper looks like the following:

/* Checks if an existing cache tag can be reused for a new association. */
static bool cache_tage_match(struct cache_tag *tag, u16 domain_id,
                                struct intel_iommu *iommu, struct device 
*dev,
                                ioasid_t pasid, enum cache_tag_type type)
{
         if (tag->type != type)
                 return false;

         if (tag->domain_id != domain_id || tag->pasid != pasid)
                 return false;

         if (type == CACHE_TAG_IOTLB || type == CACHE_TAG_PARENT_IOTLB)
                 return tag->iommu == iommu;

         if (type == CACHE_TAG_DEVTLB || type == CACHE_TAG_PARENT_DEVTLB)
                 return tag->dev == dev;

         return false;
}

>> +}
>> +
>> +/* Assign a cache tag with specified type to domain. */
>> +static int cache_tag_assign(struct dmar_domain *domain, u16 did,
>> +			    struct device *dev, ioasid_t pasid,
>> +			    enum cache_tag_type type)
>> +{
>> +	struct device_domain_info *info = dev_iommu_priv_get(dev);
>> +	struct intel_iommu *iommu = info->iommu;
>> +	struct cache_tag *tag, *temp;
>> +	unsigned long flags;
>> +
>> +	tag = kzalloc(sizeof(*tag), GFP_KERNEL);
>> +	if (!tag)
>> +		return -ENOMEM;
>> +
>> +	tag->type = type;
>> +	tag->iommu = iommu;
>> +	tag->dev = dev;
> 
> should we set tag->dev only for DEVTLB type? It's a bit confusing to set
> it for IOTLB type which doesn't care about device. Actually doing so
> is instead misleading as the 1st device creating the tag may have been
> detached but then it will still show up in the trace when the last device
> detach destroying the tag.

For IOTLB types, perhaps we could add a struct device pointer for the
iommu. This way, the tag->dev could more directly indicate the device
implementing the cache.

> 
>> +static int __cache_tag_assign_parent_domain(struct dmar_domain
>> *domain, u16 did,
>> +					    struct device *dev, ioasid_t pasid)
> 
> this pair is similar to the earlier one except the difference on type.
> 
> what about keeping just one pair which accepts a 'parent' argument to
> decide the type internally?

Okay, let try to refine it.

> 
> 
>> +/*
>> + * Assigns cache tags to a domain when it's associated with a device's
>> + * PASID using a specific domain ID.
> 
> s/Assigns/Assign/

Done.

> 
>> +
>> +	did = domain_id_iommu(domain, iommu);
>> +	ret = cache_tag_assign_domain(domain, did, dev,
>> IOMMU_NO_PASID);
> 
> there are many occurrences of this pattern. What about passing in
> a 'iommu' parameter and getting 'did' inside the helper? for svm
> it can be specialized internally too.

Perhaps, let me try it later and see what the code looks like.

> 
>> @@ -4607,10 +4623,11 @@ static void
>> intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
>>   	 */
>>   	if (domain->type == IOMMU_DOMAIN_SVA) {
>>   		intel_svm_remove_dev_pasid(dev, pasid);
>> +		cache_tag_unassign_domain(dmar_domain,
>> +					  FLPT_DEFAULT_DID, dev, pasid);
> 
> is it correct to destroy the tag before teardown completes, e.g. iotlb still
> needs to be flushed in intel_pasid_tear_down_entry()?

You are right. iotlb still needs to be there until the teardown
completes. I will investigate this more later.

Beset regards,
baolu