Received-SPF: pass (google.com: domain of linux-kernel+bounces-74768-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Wed, 21 Feb 2024 09:17:25 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Zhangfei Gao <zhangfei.gao@linaro.org>
Cc: Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
	jean-philippe <jean-philippe@linaro.org>, baolu.lu@linux.intel.com,
	"Zhang, Tina" <tina.zhang@intel.com>, kevin.tian@intel.com,
	iommu@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] iommu: Fix iommu_sva_bind_device to the same domain
Message-ID: <20240221131725.GR13330@nvidia.com>
References: <20240221110658.529-1-zhangfei.gao@linaro.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240221110658.529-1-zhangfei.gao@linaro.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?asFWpdYNUMqMT4CZwuvzfm3gTlvpsxM63sY6pRqs00VlASRUhIC6H9oZeW7x?=
 =?us-ascii?Q?mUeVXBSaQ3zgwOdB7NNl9aGMHv1v1YXhP8Z8LwYmaKGA/fC4XXdnfNc6cQPp?=
 =?us-ascii?Q?yDa3F/9E1Z8NyrhqQQxNw05PgI/TR03WI9v1uLazfBdrySlgH/TL7A9yXcVe?=
 =?us-ascii?Q?hrKfBEa/byU3yCUJq6YiN/VpoArojcTiXfcoptAMJmqq6fdqTF+HB1boNe46?=
 =?us-ascii?Q?hXyVlFRmTos0bQK2PSOzwoQQ2Pw3ORf3SrxmxfYyOtLH985G3Efxwtud6D/i?=
 =?us-ascii?Q?P4b6zY7l99c2pw9do84ixNGXGmuJcMZwdeOquXMvFSZdpG4TR/twfv9wKHGQ?=
 =?us-ascii?Q?14w38Tky/qs/Sz7y1MWAJFoxnUvLQvXV2rtT00DjlmOoiTVlMQPe84RAY7nF?=
 =?us-ascii?Q?CxmDgWK53G5XhKBxArqrQKOon9eUIXv7oL+azWXPLGq0u/LvmUHYUnI2HN6T?=
 =?us-ascii?Q?OBH+vx65O0beBP1O321F5Rf/WIHT9lHSdPFieOUxAdSDiDlrR3BmmdqoTiui?=
 =?us-ascii?Q?K2vzVJAd8zkPEXogVXTUwjiuCAgCmAJzHCYSdENX3rYWa2dEffaqg//Rdd44?=
 =?us-ascii?Q?5UyS1c67M6zRVwWP5nWc8Qt7Mk+mJJOkiC/yRi8sQ1Rdq5WN4hGiSNfQI4yQ?=
 =?us-ascii?Q?fNV90/IUrJO6VCkTyAvSYpdHycoTI25HpiTO1KBYj7lRuf3DwVcG7Z4Abrbg?=
 =?us-ascii?Q?pJxU1wFTRWBF+m2PU3T52nY4L25voY/FFI9dmsw6kjnrGtSnWiLAutnNtulc?=
 =?us-ascii?Q?7GGN9Msp/Rm1g7H+W8U2zeWyJl+NeLDzW7QqFn2vPVr5+asmyA091m+olbJN?=
 =?us-ascii?Q?mM7MuDbDcYORDvHNBd9XxzNdw+Za9B9hLNsRvGYAul8PO0OwML3Ipcd2obZB?=
 =?us-ascii?Q?OSvAPodkP+t9P/hrlws9tPCkWnvscyvuLnoagsAIinq1/PDA0CeInZYByc9c?=
 =?us-ascii?Q?i2fpzYjBlnECmWy+rDoMhMsmrKadadnjLv/7CBlJbqfqe0N0Uzb0A+xNPUyz?=
 =?us-ascii?Q?7P6Ehp5j2Q5wYZryFYnS8ADiGjuEIcrsO0bLgovMNQLITAbZNYmQkMI5WOpe?=
 =?us-ascii?Q?R/MPFXSYAVYkC8Rrx0RT03B5tw0NgTxxJlDWz3VPPvi6CL/DR3VYJJQ/n360?=
 =?us-ascii?Q?6NMaM0O4L+HreOQesaL7Ftjbt0tDK+n8cOtMkc5IJ9aZy5Z1powF1FTnXKtQ?=
 =?us-ascii?Q?3gdCZs8BaIq3zTxDPkpGwtKeT5A98+33KJWwYaG/FUgUidWn0hHhw3RANPwB?=
 =?us-ascii?Q?hL9EnsrBAkfhLBenNYsEHqSQRVchrITXX/R6Gu/yVAm9t6Fg6YdU+7Sfvjo0?=
 =?us-ascii?Q?tOzXcm7eWk2XsAU5xnktuvsUJpSCxXc43THhPuA3DeVQ+blUgePP7DC5LC5p?=
 =?us-ascii?Q?eFs0dl3XWe8lif0h/Dh4K6ursLPfJnJSp80ywllnvdlJXClpItzkM9H95edN?=
 =?us-ascii?Q?2/+IAvofEt4OeCkvrE1WI6B61AKhO/yNhUCtcxAgUWpMFWP3MivEAAUy87U7?=
 =?us-ascii?Q?ny7XtBKCfRxScYrUk5eAUrIo9pbE+zAr20T7EIeceUOtpZfWaFPVv51F2zwW?=
 =?us-ascii?Q?v/wa1EZvtpGm71DNVUlXWmVjIqHFvuzoUx+BxZrP?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8b8fbaa5-484e-4f36-57fd-08dc32df724b
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Feb 2024 13:17:25.7169
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hAd3bFFpVHz5qRvehK2S8ZF0fOO1oyHSYjbm2MQqv5qwgEofX08w341d2DUuQiq6
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9205

On Wed, Feb 21, 2024 at 11:06:58AM +0000, Zhangfei Gao wrote:
> The accelerator device can provide multi-queue and bind to
> the same domain in multi-thread for better performance,
> and domain refcount takes care of it.
> 
> 'commit 092edaddb660 ("iommu: Support mm PASID 1:n with sva domains")'
> removes the possibility, so fix it
> 
> Fixs: '092edaddb660 ("iommu: Support mm PASID 1:n with sva domains")'
> Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
> ---
> v2: Instead of checking ret == -EBUSY, 
>     change iommu_attach_device_pasid return value from -EBUSY to 0
>     when pasid entry is found, and refcount++ when return
> 
>  drivers/iommu/iommu-sva.c | 2 +-
>  drivers/iommu/iommu.c     | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
> index c3fc9201d0be..20b232c7675d 100644
> --- a/drivers/iommu/iommu-sva.c
> +++ b/drivers/iommu/iommu-sva.c
> @@ -141,8 +141,8 @@ void iommu_sva_unbind_device(struct iommu_sva *handle)
>  	struct device *dev = handle->dev;
>  
>  	mutex_lock(&iommu_sva_lock);
> -	iommu_detach_device_pasid(domain, dev, iommu_mm->pasid);
>  	if (--domain->users == 0) {
> +		iommu_detach_device_pasid(domain, dev, iommu_mm->pasid);
>  		list_del(&domain->next);
>  		iommu_domain_free(domain);
>  	}

The users refcount is not to provide for sharing of the same PASID it
is to provide for sharing the domain across devices. This change would
break that because we loose the 'dev' that needs to be detached in a
multi-device case if we don't immediately call detach_device_pasid
here.

You'd need to build something much more complicated here to allow
PASID sharing.

I wonder if this case is common enough to warrant the core code to get
involved. I suppose maybe, does idxd have the same problem? It can
only open it's cdev once because of this - that doesn't seem like what
the code intends for a non-wq_dedicated?

More like this:

diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
index c3fc9201d0be97..aec11e5cde6b0e 100644
--- a/drivers/iommu/iommu-sva.c
+++ b/drivers/iommu/iommu-sva.c
@@ -41,6 +41,7 @@ static struct iommu_mm_data *iommu_alloc_mm_data(struct mm_struct *mm, struct de
 	}
 	iommu_mm->pasid = pasid;
 	INIT_LIST_HEAD(&iommu_mm->sva_domains);
+	INIT_LIST_HEAD(&iommu_mm->sva_handles);
 	/*
 	 * Make sure the write to mm->iommu_mm is not reordered in front of
 	 * initialization to iommu_mm fields. If it does, readers may see a
@@ -82,6 +83,13 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
 		goto out_unlock;
 	}
 
+	list_for_each_entry(handle, &mm->iommu_mm->sva_handles, handle_item) {
+		if (handle->dev == dev && handle->domain->mm == mm) {
+			refcount_inc(&handle->users);
+			return handle;
+		}
+	}
+
 	handle = kzalloc(sizeof(*handle), GFP_KERNEL);
 	if (!handle) {
 		ret = -ENOMEM;
@@ -109,6 +117,7 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
 		goto out_free_domain;
 	domain->users = 1;
 	list_add(&domain->next, &mm->iommu_mm->sva_domains);
+	list_add(&handle->handle_item, &mm->iommu_mm->sva_handles);
 
 out:
 	mutex_unlock(&iommu_sva_lock);
@@ -141,6 +150,12 @@ void iommu_sva_unbind_device(struct iommu_sva *handle)
 	struct device *dev = handle->dev;
 
 	mutex_lock(&iommu_sva_lock);
+	if (!refcount_dec_and_test(&handle->users)) {
+		mutex_unlock(&iommu_sva_lock);
+		return;
+	}
+	list_del(&handle->handle_item);
+
 	iommu_detach_device_pasid(domain, dev, iommu_mm->pasid);
 	if (--domain->users == 0) {
 		list_del(&domain->next);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 1ea2a820e1eb03..5e27cb3a3be99b 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -892,11 +892,14 @@ struct iommu_fwspec {
 struct iommu_sva {
 	struct device			*dev;
 	struct iommu_domain		*domain;
+	struct list_head		handle_item;
+	refcount_t			users;
 };
 
 struct iommu_mm_data {
 	u32			pasid;
 	struct list_head	sva_domains;
+	struct list_head	sva_handles;
 };
 
 int iommu_fwspec_init(struct device *dev, struct fwnode_handle *iommu_fwnode,