Received-SPF: pass (google.com: domain of linux-kernel+bounces-109955-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <7a4777dd-2359-4bcd-839e-c2d0b5f6be14@intel.com>
Date: Thu, 21 Mar 2024 19:26:41 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/8] iommu: Introduce a replace API for device pasid
Content-Language: en-US
From: Yi Liu <yi.l.liu@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: Baolu Lu <baolu.lu@linux.intel.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	"joro@8bytes.org" <joro@8bytes.org>, "alex.williamson@redhat.com"
	<alex.williamson@redhat.com>, "robin.murphy@arm.com" <robin.murphy@arm.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>, "eric.auger@redhat.com"
	<eric.auger@redhat.com>, "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "mjrosato@linux.ibm.com"
	<mjrosato@linux.ibm.com>, "chao.p.peng@linux.intel.com"
	<chao.p.peng@linux.intel.com>, "yi.y.sun@linux.intel.com"
	<yi.y.sun@linux.intel.com>, "peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>
References: <20231127063428.127436-1-yi.l.liu@intel.com>
 <20231127063428.127436-2-yi.l.liu@intel.com>
 <20240115171950.GL734935@nvidia.com>
 <c831bf5e-f623-402d-9347-8718987d1610@intel.com>
 <BN9PR11MB52766161477C2540969C83568C242@BN9PR11MB5276.namprd11.prod.outlook.com>
 <585423de-9173-4c97-b596-71e1564d8b4e@intel.com>
 <87a2be0d-6a24-4ca8-be30-35287072dda4@linux.intel.com>
 <749b23c7-ab0e-42b4-9992-e1867fc7d4d7@intel.com>
 <20240318165247.GD5825@nvidia.com>
 <13645a9f-239a-46c9-bde2-a1d5c365df4f@intel.com>
 <20240320123803.GD159172@nvidia.com>
 <65c517a9-72dc-4342-b2f2-c3c44735bfad@intel.com>
In-Reply-To: <65c517a9-72dc-4342-b2f2-c3c44735bfad@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?STlEbzF6WW14WlVwWDZIME9IMWRzS3BCM0NpdGFOOUdERUhkQXU0Z1lWSzBr?=
 =?utf-8?B?Z3NqalJmVGRheUhVdHhWREZmSHR3a0JtTFNJVE5HcDJZL1FudUFiZGs2LzFL?=
 =?utf-8?B?R2d0dHE5ZzFXYlhMY3hGZm9XRjVvL2RTYW1CbHo2MnZPOElkZEpOZGRKNElG?=
 =?utf-8?B?eG1jeEI1b2E2WUdqMlJ2SE1SY29td2RkaDRjOUh0ZVZCTEswYStSVkNTOHZa?=
 =?utf-8?B?R1F0b1FXeFhwZGNKV3FIcE5jUllNbUlTcmNKU0hPZ2NvcFd5OE43WnRQWjU3?=
 =?utf-8?B?OExmSGdzZHluUldKMGtaWlNIWHFkd0hLNFE0VmRkRHlsMGhiYXhzVnQzbUR5?=
 =?utf-8?B?Rnh3MHdpYWtmOW4xMmJiSUgrLzZSYkl2WDB0ZjNwb1JRc1NHY1Nud0MyQmNr?=
 =?utf-8?B?Ky90UDZwZHdMRjQ0MUZYd2hZWnluM1FrMTF6T213RXVxNjdHaXBvdi83aGVY?=
 =?utf-8?B?SldLTEVBTlRwTFBiVXV1emZUQk5UMmx4bFErSmQwczA3TTYzOFNUQ3Y3dFpj?=
 =?utf-8?B?dnNyeU9Hc1J5a0RBcXpReEp5Q2d6SFViNlBjYVhVazFQL0N5eEtKQkxFdGpi?=
 =?utf-8?B?RVRRay85NncxV2VWYjJPYlU4ZTRBMWFKMElaS1lHUk50RUNPYUhXazR3bXho?=
 =?utf-8?B?eTRSU0dsdWZUUVViQXRmRWdRRmRzN0hBQUVFcjRCQS9IMTVrdUFyUStKb0cy?=
 =?utf-8?B?cUhJdXJkWGJxVUV2WGxOTGQ5c1VUNUJEaHBpOXpNYjFBS0hod0FLck9ERDVk?=
 =?utf-8?B?ODJaOFFCWUxWNVZKUXZLUFBldjc3NWdPVnhSQ0tUNTVtNjRFZmVJRmZjTE1C?=
 =?utf-8?B?OGZZazJjY1hpRzlyRXI3VjREaGlDM2VhVHZUSHg2Lzl3K2F3SjV3Vm8zcDI1?=
 =?utf-8?B?cU5TTEJsTGk4eE9ERzZRM3RmQlNDQnF1MUpLL0tIWVErd3RKam1tWWx6S0dJ?=
 =?utf-8?B?UHhhelJSRnFXVXRZUC80S2szQUNTTVpuZ3EvSEhJQVVmZW1yVElHU21pdElq?=
 =?utf-8?B?MUl5S0hTeUx0QUN2Rk1kbDRuSGZGMStZTzNCZnhZWXdhZjlrQmJ4N05SaWRz?=
 =?utf-8?B?V0FqMjNhQUZxbWQ2dDJ4cHlPZUFycUt6Q1hBR201bGhWUVVXKzhnTEZwZWtN?=
 =?utf-8?B?dERsbXhWanRrWTZtSjc2cUNVdVZUbHhMZG9MVjVYY0pBZHpwUmRSYlhGRW5P?=
 =?utf-8?B?Z09JMGpQMUh6VUY4dVlBelQyM1NKcTRVbTROdVkxb1l1MzRIRjRDcFFvUkF6?=
 =?utf-8?B?SlRUOURpN05MMEI1VDFENldrazNWUCtib2M4bHk4eWVJYkY2d1QrY2czVVRv?=
 =?utf-8?B?YnVLTDlRREpQUlU2MVYrYi9vd1hoRXhHZHZtcFpoUGZ2T24zaVdTeVk1aU0y?=
 =?utf-8?B?ZzJvOVdsNEs0ekR1S3Q0QkRHUW1OZ3MzMXhSdVk0clNtWGgwOVU5cGFuaU5O?=
 =?utf-8?B?NDIwY0lVMmI2MFdoYjlPcXdjNXpCRG5Bdmt3TEM4Yk5RMWppWTMxUWtDVWUr?=
 =?utf-8?B?dUZiZ3k5TnBtaTg4Mlhsdis4MjdvNk9oNmtFZnZreW5qajRuNU50MXV5ajNS?=
 =?utf-8?B?NHdZbHFyM2VBRkF4K2lCK1BHT3BLZnpUQjBxMFQrR2cxTTYxMmZLWWpaZk95?=
 =?utf-8?B?NFdvdE9ZVkpQdUZzUXVWdlhzM2YvbkNyVUlSRFRMdFdLc1QxbWFsdC91UDMr?=
 =?utf-8?B?a1ZIVnIrTGhGalVQYXJLeFFEWGJLeFpTWWE5Z2MyRUt2V25QZnkvcytRZGor?=
 =?utf-8?B?cG9LRndZN3JBWTFEU2lZWWh1VHVqb1Y3a0tmVUtCemwzaHRVNlNoRkN1WDg3?=
 =?utf-8?B?SlQ4QlJqUldiVmZTejZBQXp1WnpwaC94VWRyNjl1cVNCMFl1dEdlVmhrOTR3?=
 =?utf-8?B?OTZ2N0ZZdnZkRmlWakhScDVYL0oxb1pNWm5sYkhESitNRENCS01saU03WUdm?=
 =?utf-8?B?WjlRa29HZXdPNVU4d1BjZmthREJ5dGRtL3VXVUxlRGVyMlJVbkhmVlhMZUs1?=
 =?utf-8?B?dk1DbXE4TmJtUzVnQWNpcStUU2N5eW9sN1N1anBlelFhMGd4WUxKSzhsbkVC?=
 =?utf-8?B?bHF1aDE3a2hmWk5DNFJKSW9yODE1SkZLMGdNV0c3Wm9FTndoRGxqVEVEYndJ?=
 =?utf-8?Q?sVnbaygoBktEcA5WFJZ4N7CLO?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f59d8caa-a06e-45e6-fb22-08dc49994dca
X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7540.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2024 11:23:16.7054
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yHj5eAQEWaT/g43vfu+irkX728v9g+xYv19gHiLzloFvn5D7+4eEI8HXPmBGeivDiYA0lgBOJFms0v/nX6lKgg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB6095
X-OriginatorOrg: intel.com

On 2024/3/21 14:16, Yi Liu wrote:
> On 2024/3/20 20:38, Jason Gunthorpe wrote:
>> On Tue, Mar 19, 2024 at 03:29:39PM +0800, Yi Liu wrote:
>>> On 2024/3/19 00:52, Jason Gunthorpe wrote:
>>>> On Wed, Mar 13, 2024 at 04:11:41PM +0800, Yi Liu wrote:
>>>>
>>>>> yes. how about your opinion? @Jason. I noticed the set_dev_pasid callback
>>>>> and pasid_array update is under the group->lock, so update it should be
>>>>> fine to adjust the order to update pasid_array after set_dev_pasid 
>>>>> returns.
>>>>
>>>> Yes, it makes some sense
>>>>
>>>> But, also I would like it very much if we just have the core pass in
>>>> the actual old domain as a an addition function argument.
>>>
>>> ok, this works too. For normal attach, just pass in a NULL old domain.
>>>
>>>> I think we have some small mistakes in multi-device group error
>>>> unwinding for remove because the global xarray can't isn't actually
>>>> going to be correct in all scenarios.
>>>
>>> do you mean the __iommu_remove_group_pasid() call in the below function?
>>> Currently, it is called when __iommu_set_group_pasid() failed. However,
>>> __iommu_set_group_pasid() may need to do remove itself when error happens,
>>> so the helper can be more self-contained. Or you mean something else?
>>
>> Yes..
>>
>>> int iommu_attach_device_pasid(struct iommu_domain *domain,
>>>                   struct device *dev, ioasid_t pasid)
>>> {
>>>     /* Caller must be a probed driver on dev */
>>>     struct iommu_group *group = dev->iommu_group;
>>>     void *curr;
>>>     int ret;
>>>
>>>     if (!domain->ops->set_dev_pasid)
>>>         return -EOPNOTSUPP;
>>>
>>>     if (!group)
>>>         return -ENODEV;
>>>
>>>     if (!dev_has_iommu(dev) || dev_iommu_ops(dev) != domain->owner)
>>>         return -EINVAL;
>>>
>>>     mutex_lock(&group->mutex);
>>>     curr = xa_cmpxchg(&group->pasid_array, pasid, NULL, domain, 
>>> GFP_KERNEL);
>>>     if (curr) {
>>>         ret = xa_err(curr) ? : -EBUSY;
>>>         goto out_unlock;
>>>     }
>>>
>>>     ret = __iommu_set_group_pasid(domain, group, pasid);
>>
>> So here we have the xa set to the new domain
> 
> aha, yes. The underlying driver won't be able to get the correct domain
> in the remove_dev_pasid callback.
> 
>>>     if (ret) {
>>>         __iommu_remove_group_pasid(group, pasid);
>>
>> And here we still have it set to the new domain even though some of
>> the devices within the group failed to attach. The logic needs to be
>> more like the main domain attach path where iterate and then undo only
>> what we did
> 
> yes, the correct way is to undo what have been done before the fail
> device. However, I somehow remember that pasid capability is only
> available when the group is singleton. So iterate all devices of the
> devices just means one device in fact. If this is true, then the
> current code is fine although a bit confusing.
> 
>> And the whole thing is easier to reason about if an input argument
>> specifies the current attached domain instead of having the driver
>> read it from the xarray.
> 
> yep, will correct it as a fix patch.

Hi Jason,

It appears there are two solutions here.

First, only undo the devices that have set_dev_pasid successfully in
the __iommu_set_group_pasid(), so the problematic
__iommu_remove_group_pasid() call at line 3378 [1] would go away.
This also makes the helper more self-contained. Draft patch in [2]

Second, pass in the domain to remove_dev_pasid(). Draft patch in [3]

Either of the above two should be able to solve the mistake you mentioned.
BTW. They are orthogonal, so it's also possible to apply both of them.
Which one is your preference then?

[1] 
https://github.com/torvalds/linux/blob/b3603fcb79b1036acae10602bffc4855a4b9af80/drivers/iommu/iommu.c#L3378
[2] 
https://github.com/yiliu1765/iommufd/commit/bb83270767ab460978a3452750f5e032b43e59bf
[3] 
https://github.com/yiliu1765/iommufd/commit/bd270b7b6619b8ba35dfaf9870df8728e370163f

Regards,
Yi Liu