Received-SPF: pass (google.com: domain of linux-kernel+bounces-98224-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <c831bf5e-f623-402d-9347-8718987d1610@intel.com>
Date: Sun, 10 Mar 2024 21:05:58 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/8] iommu: Introduce a replace API for device pasid
Content-Language: en-US
To: Jason Gunthorpe <jgg@nvidia.com>
CC: <joro@8bytes.org>, <alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <cohuck@redhat.com>,
	<eric.auger@redhat.com>, <nicolinc@nvidia.com>, <kvm@vger.kernel.org>,
	<mjrosato@linux.ibm.com>, <chao.p.peng@linux.intel.com>,
	<yi.y.sun@linux.intel.com>, <peterx@redhat.com>, <jasowang@redhat.com>,
	<shameerali.kolothum.thodi@huawei.com>, <lulu@redhat.com>,
	<suravee.suthikulpanit@amd.com>, <iommu@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>, <linux-kselftest@vger.kernel.org>,
	<zhenzhong.duan@intel.com>, <joao.m.martins@oracle.com>,
	<xin.zeng@intel.com>, <yan.y.zhao@intel.com>
References: <20231127063428.127436-1-yi.l.liu@intel.com>
 <20231127063428.127436-2-yi.l.liu@intel.com>
 <20240115171950.GL734935@nvidia.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <20240115171950.GL734935@nvidia.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZUkyOHZINjZRQmsyQzJBQzVuZS9PWXJMNnRRbTVCYmExQy9DaVJ0SWd0UmRx?=
 =?utf-8?B?NnNuTDRKckdOYTRWbEVJSTZFbGpodU9OS3lyRzBhVjNaWEpPU2dqYUNnQVpi?=
 =?utf-8?B?NTZVVzZFKzdPdW9tLzV2SHpLN29BR2FJakFvNVk3LzYrWHdtLy9TNWdmekVB?=
 =?utf-8?B?YjgxVFpRcHdkb3lhajBKa01JZEUwTFdtNHRCS1FjdWxVVVdLT1J1SlRsWW5B?=
 =?utf-8?B?WnVwa2NMbzIzUnYzRXFqNkhTR1JCWlFUWHVPMno4T05tZGltdEFuc3QzS2Mv?=
 =?utf-8?B?bGQwQ21HQko3TkVxRFNRMmxxb3NaRU5sTnorSGZObHQvZEJZS2craE12Y2Mr?=
 =?utf-8?B?ZVh2ejRpYUltNW9WcGs3YVcvVDFtVnhUaVpXMlF3Y0p1dmQycHRLQ2ErQmd1?=
 =?utf-8?B?S0MzVjFJZm1HNEFXZHFmVnZGZmZjUTA5eHRPTE0zRm5BTGZUMnRMZmVoaU1X?=
 =?utf-8?B?eDdYOGZqZWNrQW5PZzdrMTExeS9ja3BFcUdqbkNRMkRMZ2V4TmtrQVBDV2dv?=
 =?utf-8?B?ekNZVDVQeFFSa3h5a0FQOVZRKzZhL0NQeUpFRWJWKzJTdzBPVXpnT2J5WXpp?=
 =?utf-8?B?THFWdW1sbHlyMzB3N2JLWis0THNxSXNNMy9aVjZPeUptRE85SEV2TllnVWh1?=
 =?utf-8?B?dWJwNkhGc0UyeStKMWRxSTVMbWNVcmYvbVk1eTVhVHAzY3JNUzRJdjNMUHND?=
 =?utf-8?B?dUtEdFZEVHRDSjJzQjU3NUZlMnd3K3NRL0l4aWM2L1ZDOHRVTkZiN2t2Wkdz?=
 =?utf-8?B?QVNJdzZEWTlKdkgyUDNYSUZXWjAxTzNmNkpvT0hwNlN6bGMwM3gwdGE0Q0ZN?=
 =?utf-8?B?UjJja0RtQUlIdkZWa1JCQ2tWcjhnT2xqOGJQOXVxeHZaOE9YcW5WcjZBTzE4?=
 =?utf-8?B?anZuV0tpOC9ha1VWM0paVW1ZejZhS3JRSlNxb2hiUEVxZm5YM0RrTGtMeE1r?=
 =?utf-8?B?UlBaSElrT1NHY09jTitENWl3MU51cE92YjUvamRhYWp0WmdveldZZ1VWdS9o?=
 =?utf-8?B?bWRzVXh2YjJXdlBGdU9BZVpJYkZUbkNDcjd0TXN0MVEzZ1o4THhKbDVZaFBP?=
 =?utf-8?B?MkRzTXQ1aXhFQnNEVkJaOENTS3UvVkNQN2hvV3pZWlMwdG9yK2tPREJqZUNh?=
 =?utf-8?B?NDdPdVRKaUdJazRjckNOc01jY3RROVhoZHByZWJSaXM2bHU2ZjU1VXIxVE8w?=
 =?utf-8?B?SXlVZWtJaE5KaW5WM2tGMlpHb2RlOU9ucDhWbFNLUnRua1AwYlZuSVB2RnA4?=
 =?utf-8?B?Z3E5UzE0TnhNTHdtTnR5ZURGdGtnN20wajRNUWxEZUlMSVRNemJRNExwYndJ?=
 =?utf-8?B?UU1lTXo4cUgvZmNWajRGTEN3TnE2Q3FXS3ZBeEhSMjVBVGZJcFp2NGt2WlNz?=
 =?utf-8?B?UndzZWJZOUNVblIwZkJlQzA1RTRPY2l3MXFFbGtQc2lubzY4cXJGWk0zRkRY?=
 =?utf-8?B?eEFvdTR3OC9kL2wxTXBHWDBnNEVyVDY2R3JWajVQTVA1cGJLR3NaZW0zRnZo?=
 =?utf-8?B?QkhoVVJTVHQvbk14WlJ0OXpBRUVtQ3BZU04yZG04azRnWE5XWmVRVlZxempm?=
 =?utf-8?B?dkRCRWhMbXdTZjd4OHZKOEo2a0srMXlQODRZelRhSkphbzc5KzhkVUZlN3pF?=
 =?utf-8?B?NEhJOXFzNFRhK28xN2NCUHJtWGYxbWdMNDl1MGFMSm5KRzkxRWo0M2w4aXQy?=
 =?utf-8?B?d04xbjZieUNCU2lrL0JkbWwwODFGWlExSW5QS285Zy9JOWVockVsRlBuL3dZ?=
 =?utf-8?B?RVZPMzl1Tmk4SC85bWRGR21UMFRqSVVheVZLYWw1SnB5ZHhsWEhOV2Vqc1pE?=
 =?utf-8?B?V2NPMWxLZFpjbVJTdDJSbVc4RVRkWC9ZeUFnNUtJQ28zTG5DZG5ERjZ1Tm53?=
 =?utf-8?B?aEJVWktXRzFZbDdWYU1ueDdxNVE4SlE3YmZrc1NrcTkvTUhnZGhIbUpkaFhN?=
 =?utf-8?B?WDQxTEswMzBuQjdER3hFcG1MY1FyRnhaRldCOUwzSS9uNHN5bktJYWpsWHIy?=
 =?utf-8?B?QjNNaU5vVXg4bUhGdEsycStLNTU1WWlzUVU1OGpuTTZ4UnduYmxramRFTlYw?=
 =?utf-8?B?NGovYkhMcnRLL29mZ3RaMXluNCtBeTdFTGc4WndJTFM4NGtlaW1lWVpLTUJY?=
 =?utf-8?Q?oSa0a5LZwmSe5bDqzQDpWaJY3?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 48ce4173-23b8-4079-cf18-08dc41025d67
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2024 13:02:39.6101
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KqwJAyOr6aWAqe60TYEU9+39XHpcu1batpo8MaGU9TIi4monK8anywH4/YJUH92ZQRxqZryavsLloGIAQCJWgw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7107
X-OriginatorOrg: intel.com

On 2024/1/16 01:19, Jason Gunthorpe wrote:
> On Sun, Nov 26, 2023 at 10:34:21PM -0800, Yi Liu wrote:
>> +int iommu_replace_device_pasid(struct iommu_domain *domain,
>> +			       struct device *dev, ioasid_t pasid)
>> +{
>> +	struct iommu_group *group = dev->iommu_group;
>> +	struct iommu_domain *old_domain;
>> +	int ret;
>> +
>> +	if (!domain)
>> +		return -EINVAL;
>> +
>> +	if (!group)
>> +		return -ENODEV;
>> +
>> +	mutex_lock(&group->mutex);
>> +	__iommu_remove_group_pasid(group, pasid);
> 
> It is not replace if you do remove first.
> 
> Replace must just call set_dev_pasid and nothing much else..

Seems uneasy to do it so far. The VT-d driver needs to get the old domain
first in order to do replacement. However, VT-d driver does not track the
attached domains of pasids. It gets domain of a pasid
by iommu_get_domain_for_dev_pasid(). Like intel_iommu_remove_dev_pasid)
in link [1]. While the iommu layer exchanges the domain in the
group->pasid_array before calling into VT-d driver. So when calling into
VT-d driver, the domain got by iommu_get_domain_for_dev_pasid() is already
the new domain. To solve it, we may need to let VT-d driver have its
own tracking on the domains. How about your thoughts? @Jason, @Kevin, @Baoplu?

[1] 
https://github.com/torvalds/linux/blob/master/drivers/iommu/intel/iommu.c#L4621C19-L4621C20

-- 
Regards,
Yi Liu