Received-SPF: pass (google.com: domain of linux-kernel+bounces-31008-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <cb52ad1f-822f-4da6-9753-691454565b26@intel.com>
Date: Fri, 19 Jan 2024 18:15:30 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 3/8] iommufd: Support attach/replace hwpt per pasid
To: Jason Gunthorpe <jgg@nvidia.com>
CC: "Tian, Kevin" <kevin.tian@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>, "baolu.lu@linux.intel.com"
	<baolu.lu@linux.intel.com>, "cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "nicolinc@nvidia.com"
	<nicolinc@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>, "peterx@redhat.com"
	<peterx@redhat.com>, "jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>
References: <20231127063428.127436-1-yi.l.liu@intel.com>
 <20231127063428.127436-4-yi.l.liu@intel.com>
 <20240115172430.GN734935@nvidia.com>
 <BN9PR11MB52761349DFB5DAD2797C3EBF8C732@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240116125756.GB734935@nvidia.com>
 <BN9PR11MB52763DDDE39C211E761A05168C722@BN9PR11MB5276.namprd11.prod.outlook.com>
 <88e46f6c-4d64-4357-be2a-833797e6de15@intel.com>
 <20240117125625.GF734935@nvidia.com>
 <459d6a3a-0ad9-4980-be37-103211e927c2@intel.com>
 <20240118133857.GK734935@nvidia.com>
Content-Language: en-US
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <20240118133857.GK734935@nvidia.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bFFWc25WOWdpWDRhNW9adS9uQkQ2WTVERER3K3NKUkh1N2RXaW1POXNnUWlW?=
 =?utf-8?B?NUUyd0tXUHBZN1dEejF4UFErQjErR2NDaUdBVG15bjFZVTFBU2IyVUphSXJT?=
 =?utf-8?B?WHhFbDdQdTJDWnBLRVJ3ajFXK2pRTm5UQndKVy9nZ1dRVit3SzVteDFyWm5o?=
 =?utf-8?B?YWVrQW5LNmlla2lJWGdObnVmWTNWa2dzZnU5cjRQY1Q4eHJkbXBhcFJtSWJI?=
 =?utf-8?B?VXR4Mkt3M1Z2YTJWS1BERy9FMzFMVEREdThsQzY0TWtCQVpFVzBHNkhwdnB6?=
 =?utf-8?B?UGVtbHNGQ0JrT1BLbExibTgxL3VHNDNuNVY3dHQrTy9VWmhxN2pGVWRxbzNj?=
 =?utf-8?B?dFNSdXcydU5EVXJzVXQ3UUdjWVdJR3NQN2ZDMS9RVG5rWWlnNlFOMjdUTFgx?=
 =?utf-8?B?RUFlSFduMnkwbWdrMXA0S3Q5Vk9KRDBDT1RRejF1SkNzWGx3L2ZUMjY1ekpC?=
 =?utf-8?B?SE9jL2Q5MTMzYUh4Zm9uR09oMytjRmFrTEpVK3F1SjVqV2duWkNtZkdRYS82?=
 =?utf-8?B?S0ZzV1RraXJ6d2JWbjFkNi8vSVMvbUxRRjFVTWwvam1wN2ZwUU9JMUQ4Z0Fy?=
 =?utf-8?B?bGJMRkZHbmFyTzFvYlJCYS9ubmpHV2RjTS9zdklpQnI0bzNhZTgxYVpGcnpH?=
 =?utf-8?B?MVV4U2dUTU1ROTJNY3EzVHFCT1VCcHFRSnBNM3RDQzZCdVJNZ213WmZKb1Nx?=
 =?utf-8?B?Y3Q4aDdtY0xVNXROYkNKSytvcnBpU3paRUIwV3M3ZVlPaFcxUk1RaUJqcm9s?=
 =?utf-8?B?QXpGTEpmWmRubkFickhoSWNlYjQxU0NiVko5bWt1MTVTOVdmU0xMZVZxRkIy?=
 =?utf-8?B?UXdrVTYrcUxyS1FVVGVld2lVdXdqc1Zya0hRV1FGVGRheDFDeHltYlMxcWRV?=
 =?utf-8?B?YXl0SG4vTW0zWnBlbjd3K3FkSWNtSDM0NDdkbnhuczUrOFl4WFVsdFlnVEFH?=
 =?utf-8?B?RUkrSVBOdUpnQnphN1NJbkVkTU9mY1lxdC9LbzZRWmpyc3JEeEk3Ylh0bFgw?=
 =?utf-8?B?TDRhdVRwVEZQVmJZYmlLM0JJMlRtKzJZL0h1Z1dKVDBhdUFMU0sxN2p2dzNW?=
 =?utf-8?B?bGxvVlFFYmxLeFlYUG5kc3ZJTFcxaStybDR1bkhmUENmNWxBUmRmZ0VOOFBJ?=
 =?utf-8?B?dmlGbTFGUDdVL1B0VlpMZUlPK1UzZFdwcTVzb3hOOWNPYUQvaWNwaXUwdHZa?=
 =?utf-8?B?Q3lkWWVDSEFYRWIvMDNVWXR6QVRZKzMxcEphT3BUTUxyMDFrTE5vOWI2N2NM?=
 =?utf-8?B?TjBDdnB3L3U4ZFdFbzFMUUJreXByTFhrZW9McTJtak5WTFdKbjZUaXNmNnNC?=
 =?utf-8?B?ODVmR09OSXgvZzZVM0JFOHpHa3d6MG5yckxXRSs5VXA3bnl5NHBIN2x4bnFS?=
 =?utf-8?B?L055dWczYzNXOTkxT0NjM2ExYUhLQXJ3Q0NxdGpmb2JLQUNVVkdUeitYWTBo?=
 =?utf-8?B?TWRtalB5TG1ZVFFrenIvVjhtOFpTRHp5TDdkcXhNclV4SjJHMzhDenp5QjZT?=
 =?utf-8?B?VjRmUEpaNHJiQzVXQldDWThPSnlPM2t0TE10UUtBZjBXdVRVUEo2ekFGR25z?=
 =?utf-8?B?RmFoemcrWUUwS0pQb1A2Y09heUNYdW1CT2xWK1lqa2hTR0Z1MlZiNU9xdTNa?=
 =?utf-8?B?RmlPampaZCtqaVhBMDZrZ2UwRytPWE1yME8vd25oVzdxSFQ0SDg1eVlMTWpG?=
 =?utf-8?B?QWhQYzZSbStXZk9hZ01SSmNrWmY5dFdmTDhDMEt5Y01iZGU3OWVHS3UxTE5N?=
 =?utf-8?B?TEJmZnRBbTV6Q2Z4WU1DU0xYQlRpWDJqWkxTUXdkSVFMeGY3eHFGZGlUL2xm?=
 =?utf-8?B?bXI1WWpIT1l3L3VRZUsycFNDVnY2NmFnR3ZSRDlOczBkMjRzZndUMU5SUExp?=
 =?utf-8?B?NHkvbGx3QzIrNE9HSFF4d2ZaWkl4Sk9UR2l6Q3N4V3pweVR5TnNMZmMwU3dJ?=
 =?utf-8?B?aHV2WThWWjFEdjZia2pmb1pZaVR2Y1Fjd1BpaDBoVEpSZnV4NkoyNjgrTzN5?=
 =?utf-8?B?dDZ1dk1HcUo0RW8xZmM3alNOclpwMkV4N2d0MFZGL3RhN0ZQWnJXNVFZajZC?=
 =?utf-8?B?RHljTEdva25KNFgrUGllQnhRd1hYbDZDVWt3dHBPVG14ZjBzR09TRFYzVzJm?=
 =?utf-8?Q?p3scBnhwO6+IAL2IZVm0//nO6?=
X-MS-Exchange-CrossTenant-Network-Message-Id: da3193f5-459c-40c2-0314-08dc18d72562
X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7540.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jan 2024 10:12:30.7101
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QGSJfwXifyKXWTP0AN6Q0ptyBse/ea1cMpA5lEK1WxprCL9MbC5nVZpY7zbfTXeMIJ8TQb4n7EkK0k+gS/+BBw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4660
X-OriginatorOrg: intel.com

On 2024/1/18 21:38, Jason Gunthorpe wrote:
> On Thu, Jan 18, 2024 at 05:28:01PM +0800, Yi Liu wrote:
>> On 2024/1/17 20:56, Jason Gunthorpe wrote:
>>> On Wed, Jan 17, 2024 at 04:24:24PM +0800, Yi Liu wrote:
>>>> Above indeed makes more sense if there can be concurrent attach/replace/detach
>>>> on a single pasid. Just have one doubt should we add lock to protect the
>>>> whole attach/replace/detach paths. In the attach/replace path[1] [2], the
>>>> xarray entry is verified firstly, and then updated after returning from
>>>> iommu attach/replace API. It is uneasy to protect the xarray operations only
>>>> with xa_lock as a detach path can acquire xa_lock right after attach/replace
>>>> path checks the xarray. To avoid it, may need a mutex to protect the whole
>>>> attach/replace/detach path to avoid race. Or maybe the attach/replace path
>>>> should mark the corresponding entry as a special state that can block the
>>>> other path like detach until the attach/replace path update the final hwpt to
>>>> the xarray. Is there such state in xarray?
>>>
>>> If the caller is not allowed to make concurrent attaches/detaches to
>>> the same pasid then you can document that in a comment,
>>
>> yes. I can document it. Otherwise, we may need a mutex for pasid to allow
>> concurrent attaches/detaches.
>>
>>> but it is
>>> still better to use xarray in a self-consistent way.
>>
>> sure. I'll try. At least in the detach path, xarray should be what you've
>> suggested in prior email. Currently in the attach path, the logic is as
>> below. Perhaps I can skip the check on old_hwpt since
>> iommu_attach_device_pasid() should fail if there is an existing domain
>> attached on the pasid. Then the xarray should be more consistent. what
>> about your opinion?
>>
>> 	old_hwpt = xa_load(&idev->pasid_hwpts, pasid);
>> 	if (old_hwpt) {
>> 		/* Attach does not allow overwrite */
>> 		if (old_hwpt == hwpt)
>> 			return NULL;
>> 		else
>> 			return ERR_PTR(-EINVAL);
>> 	}
>>
>> 	rc = iommu_attach_device_pasid(hwpt->domain, idev->dev, pasid);
>> 	if (rc)
>> 		return ERR_PTR(rc);
>>
>> 	refcount_inc(&hwpt->obj.users);
>> 	xa_store(&idev->pasid_hwpts, pasid, hwpt, GFP_KERNEL);
> 
> Use xa_cmpxchg()

sure.

-- 
Regards,
Yi Liu