Received-SPF: pass (google.com: domain of linux-kernel+bounces-29949-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <459d6a3a-0ad9-4980-be37-103211e927c2@intel.com>
Date: Thu, 18 Jan 2024 17:28:01 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 3/8] iommufd: Support attach/replace hwpt per pasid
Content-Language: en-US
To: Jason Gunthorpe <jgg@nvidia.com>
CC: "Tian, Kevin" <kevin.tian@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>, "baolu.lu@linux.intel.com"
	<baolu.lu@linux.intel.com>, "cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "nicolinc@nvidia.com"
	<nicolinc@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>, "peterx@redhat.com"
	<peterx@redhat.com>, "jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>
References: <20231127063428.127436-1-yi.l.liu@intel.com>
 <20231127063428.127436-4-yi.l.liu@intel.com>
 <20240115172430.GN734935@nvidia.com>
 <BN9PR11MB52761349DFB5DAD2797C3EBF8C732@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240116125756.GB734935@nvidia.com>
 <BN9PR11MB52763DDDE39C211E761A05168C722@BN9PR11MB5276.namprd11.prod.outlook.com>
 <88e46f6c-4d64-4357-be2a-833797e6de15@intel.com>
 <20240117125625.GF734935@nvidia.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <20240117125625.GF734935@nvidia.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cjRVdnMyM3VmUnlZT1A5bDh4WHhaVEN2MWw3c0Q3c0hvdnJSV3F2RnVoNHl4?=
 =?utf-8?B?SjllRzE4WjRmM1hidFl4K3pZZk5vZko4NjV3WTVJUWpmUDFpZWpFaDM2VGNt?=
 =?utf-8?B?WVY4TTJ1cnNGZ0FyQ3lONjN6OVNRYXFLNUpOMnR6aW1QNktLN0w0RnBDcDRR?=
 =?utf-8?B?a1VvQSt2RGx0eDJuVmVrNW16bEl1Qnl3VnVNOU4vb0UvcXI5SzVrWEFra0Nk?=
 =?utf-8?B?eDZsV0psL2dkZ2t0R3hsanVBTlBlMEFzZHBlN0lnOVphZVk5TFp6bVRrUnQ5?=
 =?utf-8?B?ZkpZdW82WUpaRFptRlR2ZUpucUhkcW9XUWQ3ZUZFV1U1ZnptOS9xdzBkNkdI?=
 =?utf-8?B?NjZQa3RRS3Q2K2pqYUNKSWlwZ3RrbXNrSTB2ZWQxM1FNR2xXU09udm0xbHll?=
 =?utf-8?B?ay8vamZFZnUwdnM1RHgzN0o2RXpVeTZIRHNNZGw2MUYzU0VLVVl2V0g1ZUhR?=
 =?utf-8?B?aUdURVJhbGhMMTZTWVl4YzRxcy80NlpaMk9SMEFWU0hEdG1XTkhzRHkvazU4?=
 =?utf-8?B?YkQ1WTRsZGl1eThtc1VyMmltN0psQWRPK0xpWDdCcmpxaTRiN0UrLysyQjdi?=
 =?utf-8?B?MkdWemZwRWxRM3U0cENqOGd2bU1zV213eXUyZHRrdWFjc1pEWWUwU284YW5G?=
 =?utf-8?B?ZHdqRG4raTZMaFhqdW9MVlNkcEJMbVJ2UHdNVEJvLzdNMlk3NitsUWdQZmg1?=
 =?utf-8?B?WXZuMWRyS2srY0tQSFA0Mk9KRE1CK0loOW5QZkpkbUw3SXArd3ZiTzZKNVpT?=
 =?utf-8?B?NFlPdThzTzQ3ckN6M21kVjBibXp2dEx2TWVXWm9qRlY4bytHUml4cVhqY05t?=
 =?utf-8?B?WDVTOFp1R3RFeHk0YUlkUDIxRkVoR0xaWFZ3RWk2RUhXaGVLcTZIZUFNckxB?=
 =?utf-8?B?a3BvTURMZVcvcXIrNkhvdzU5RGdYcWtNWTVQTkg3eW9lSGlCS3JoQ3BwOWg0?=
 =?utf-8?B?blR2RGhJLyswQjNBKzlhbHlMZmZXbTQ3dUxSRjFSTVVpalhaSVRGNVU5ZkxQ?=
 =?utf-8?B?MHF1Rmx6VWlRMUVRUzFPZml2SE81amZPVmo0UEpla0ZtL3Z1RDJnT3FqOXpz?=
 =?utf-8?B?TnNUeFZWdjhnYWtTTlBaRldVSGpTSnB2UkhpS2hFdUFqOHpnTmI1RzNWeHIw?=
 =?utf-8?B?NXQ3dWwwT0xTcnNiejRPWmluemJCT3FjOXlNVU1XWEV2VVF4TTl0eU1HUWRC?=
 =?utf-8?B?WjNRK3c1M3J1WHRuUUtIOHE2KzhXN0VXU3N3NCsvRXhRSXBQbkhLU2dnQTI3?=
 =?utf-8?B?ZEdIa3pEVnl4V29DYWt3NHhjVWhZeHBwV1M2TFhUekM2bFoyVmxiWEFoVFVZ?=
 =?utf-8?B?U05DeFQ0ZmtQZmUzVDZpM0Q5S2Z4bjVDQmVaZzYyUzVWd0xUTmJzT2xHTlVr?=
 =?utf-8?B?UXYzNTVCdkl2R1hZQ29WcFFGT0pHeTFnTEgxMjFaL2NybHhnZHE1ZzZOSmRy?=
 =?utf-8?B?b1Rsc1ltbG9WdEVqS05XblJlTmN0cWRnQ09HZUtTdGN0ZWVlbGdqNlh4dlNV?=
 =?utf-8?B?Um9GeUc3a2NpYjhuVzBWdUJhdjBYZldDbnQ1TS93QjhwUWdBS0dhNnRLVTYw?=
 =?utf-8?B?NFk4OTI4aDAyRU9sUE9sYlBSR3ZPTHd2L3U3OEJhTTc5QkExaXVnNEpWV3pp?=
 =?utf-8?B?WUVCajJ6ZXEzZlhZeXBoME1nLzg0eWp1aHl3eUhzQlJHUGpFcmg3QzJyeFNO?=
 =?utf-8?B?c1dBNFRpUmZ1Zi9ENEZGcUdwUnp4ZDhYR3BKN0tMaFZXY2YvM05pb2ZhbXY5?=
 =?utf-8?B?WkVMckRTbEd1RkJFb2RwRGQ2WWdVd3JaZ1RadkdVYk5pWElUdndINnByQjVL?=
 =?utf-8?B?eEJ3NUNTdU9FNzVYVmR6OER6dTErYlFlZEpCQ0FoV05mTFlZY2EvVG9rZXVN?=
 =?utf-8?B?bEZSc3VWMlhVRTgyZFBWdG9mSHcyKzZramdwdE9YUHVyZ21yMWU2d0dFUVVr?=
 =?utf-8?B?OTBMdk4yd0ZhMkdWOXYrVGVONU9EUk9BWk1ST0N0WU5heFRxRzBjS05ncnRa?=
 =?utf-8?B?SXg2MkgzMnFXTXdJaVJOZ3hEVkJJRGdlQlAva1FuemFkcGtyR3I3MHUzdG16?=
 =?utf-8?B?blM5R2xqWFNUK3dqZkVhbnRMT21kUXl6OVA0WXNBbUpiM0Z3eUIzNkVGK3Z6?=
 =?utf-8?Q?IACSxzWBsiy7Fx01NJ+AH6xC8?=
X-MS-Exchange-CrossTenant-Network-Message-Id: cd8efee1-222c-4e8b-b102-08dc180760dc
X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7540.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jan 2024 09:25:15.1539
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iug4cia1Or0+v4m8Hu60YBRrxQjQWNDcduz8m+pSwsQvZfnuccv6GoxY1tdXPaQgsnBIeLTsdaqA1AeA0UyLAQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR11MB8358
X-OriginatorOrg: intel.com

On 2024/1/17 20:56, Jason Gunthorpe wrote:
> On Wed, Jan 17, 2024 at 04:24:24PM +0800, Yi Liu wrote:
>> Above indeed makes more sense if there can be concurrent attach/replace/detach
>> on a single pasid. Just have one doubt should we add lock to protect the
>> whole attach/replace/detach paths. In the attach/replace path[1] [2], the
>> xarray entry is verified firstly, and then updated after returning from
>> iommu attach/replace API. It is uneasy to protect the xarray operations only
>> with xa_lock as a detach path can acquire xa_lock right after attach/replace
>> path checks the xarray. To avoid it, may need a mutex to protect the whole
>> attach/replace/detach path to avoid race. Or maybe the attach/replace path
>> should mark the corresponding entry as a special state that can block the
>> other path like detach until the attach/replace path update the final hwpt to
>> the xarray. Is there such state in xarray?
> 
> If the caller is not allowed to make concurrent attaches/detaches to
> the same pasid then you can document that in a comment,

yes. I can document it. Otherwise, we may need a mutex for pasid to allow
concurrent attaches/detaches.

> but it is
> still better to use xarray in a self-consistent way.

sure. I'll try. At least in the detach path, xarray should be what you've
suggested in prior email. Currently in the attach path, the logic is as
below. Perhaps I can skip the check on old_hwpt since
iommu_attach_device_pasid() should fail if there is an existing domain
attached on the pasid. Then the xarray should be more consistent. what
about your opinion?

	old_hwpt = xa_load(&idev->pasid_hwpts, pasid);
	if (old_hwpt) {
		/* Attach does not allow overwrite */
		if (old_hwpt == hwpt)
			return NULL;
		else
			return ERR_PTR(-EINVAL);
	}

	rc = iommu_attach_device_pasid(hwpt->domain, idev->dev, pasid);
	if (rc)
		return ERR_PTR(rc);

	refcount_inc(&hwpt->obj.users);
	xa_store(&idev->pasid_hwpts, pasid, hwpt, GFP_KERNEL);

-- 
Regards,
Yi Liu