Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 10 May 2022 12:23:30 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Lu Baolu <baolu.lu@linux.intel.com>
Cc:     Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v6 08/12] iommu/sva: Use attach/detach_pasid_dev in SVA
 interfaces
Message-ID: <20220510152330.GG49344@nvidia.com>
References: <20220510061738.2761430-1-baolu.lu@linux.intel.com>
 <20220510061738.2761430-9-baolu.lu@linux.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220510061738.2761430-9-baolu.lu@linux.intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9hPr+psUlkUgXgMNl6vvbxl9+dC92QB+XfR3E/B0gGW/ddIIW7nAfgcVF+kn?=
 =?us-ascii?Q?XoEZrkS89x4goNFRkbFmDA6V0Qrg/FE4qdrn9lkGTrce1vBcnFNUJkO7V5cO?=
 =?us-ascii?Q?+OVnKqwViwCtdotitG9PPT+u9A7igKri64tQniStUeuVg+phdFKP9tol22MO?=
 =?us-ascii?Q?1Zx0ZmaO2GnaE+Bp948hzgseJv+X9JDV3SLPJ61ctBNzxKTS2oycNdnrlG78?=
 =?us-ascii?Q?jSAADRRj2j3OniCRC1j80R0d4ofbYni/jNvgFW5EvcRl3p9rKcumsPJ3iL1K?=
 =?us-ascii?Q?kLO8Losm8Y9lzyBjx4cXJUVQ1UfZ3mJF0q1py9YiUC0tpfA0F7l5euaefZSB?=
 =?us-ascii?Q?N5S2SpEEkxQ50iHkMrvjzE3x9UldsThCJvPeyZS5VKGnSauSCjPTM9YeHh1z?=
 =?us-ascii?Q?PJ2vgappWkgXNmon8u2UweBIklOyafzZziOwZsN7XQcxHKkMng/Rss0blT+E?=
 =?us-ascii?Q?yAUrlR/04TR9yeZQIjOS2d7KQv+DZysDtotAXFZfYwyz77m/lXQQjCB6sgra?=
 =?us-ascii?Q?0Iv5XXLxV/N2IwzUf4nZM/6bca4R3mNm9xxHzKdUBsmNMcuzJrlG2cFEghbU?=
 =?us-ascii?Q?IqE4YURTaiURCgebbdiGv3bLhYFIsrl1hc+V2iGDOIOmgT7XaXqiKv/XbHy4?=
 =?us-ascii?Q?UZSeaBK1Mz/Ps/4UOSLhnG1JgcTJQSyBSNOXKahyf80NQoE99Yhk+4FgTZms?=
 =?us-ascii?Q?tUD1kAfpM9c1nQNg7ykFabF6Wcl3/xcQLvW8IzY4jDJWwLPMTC017sQV+yWd?=
 =?us-ascii?Q?Iv1aGKPGnQWvrg5vbcDuLxpWf4n8yoF5Q3K/qJ8p2mI+mnfG6JQbCnQNmoyL?=
 =?us-ascii?Q?pCu7XKK42DrFZikr9Um9pQyqji6g061W160aX/lSikylSGRWTEVzfZ4YtdXQ?=
 =?us-ascii?Q?wUDKDmkKsh+NvvfhVOC5YFFVdIZplK7206PopnyztzmgiF/+EtKMN5Yo20kx?=
 =?us-ascii?Q?pdojqHReE7yS+pvgKfaYNFTZQlFJNlBvAPWQzahNGrQUiAUisLoNVgB0k5oM?=
 =?us-ascii?Q?Q/ezi3Ehz2mp8ZlMAc/3C9C89osZsXatCYwMpNUP+u2x+BYFEYogbRlxDx9Y?=
 =?us-ascii?Q?AKcYKn4UbdrILKL47nVIjrFeuiJejuTkUXRchZ13jdtfa/N/FStAFz/MwYUr?=
 =?us-ascii?Q?TEKe6uOgC1zfteL9f8TYjAD1YUpJb/EWNdTWSLcpoPIbPmGfQlAgdyhPjCka?=
 =?us-ascii?Q?jjjeBZDZrHaVL3vQCeHwFCtyTNVO7U5d9yV+BSm+RZzcYzlXHe9mU7F6BG/f?=
 =?us-ascii?Q?MRH/j9gm6pXSqEi28fPPRJsH75fqsNyyJ0niICSLxIRsBB+E/noRhcoi5GiW?=
 =?us-ascii?Q?zY87Vy0vATMS6bIF3Tozko0v1R3TrizoCMVif+4hczrxlON7o11uw6H89jXV?=
 =?us-ascii?Q?VoZa1KSoyDY1ffaM3+aPS/CRsmA6lR4ZFKdlTr2/xe8gOZXPvawNhotcADkm?=
 =?us-ascii?Q?evUiGeW6znhirh9yJ10ml5HhWedZ/YA29J3FvbzonRz15pgiTZmb4QrShHAY?=
 =?us-ascii?Q?L43ew/zfwtILemfLI2tVUwUTWhz/QOJwljatY60isjk85VC6bGON2lyog+nZ?=
 =?us-ascii?Q?tbWFPhISZNhTecJZGzsqgEVrLKqGUSATdpKg9gKV1Z6rjiTRMy3OdCuHQqR5?=
 =?us-ascii?Q?zErVbQwa1krVolJWNzrl2mWc15HrLCrStapttzFrI91073KyC825qBak4UcX?=
 =?us-ascii?Q?omf8kL1uOAcW8zAHAHPVYDJb0gLQxnGPLIoYL0kkfQYSuQ6f+jrmb37BKcMP?=
 =?us-ascii?Q?Xyna2sZYXw=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d21d48eb-8269-4bb9-328d-08da32990a5d
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2022 15:23:31.3017
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8MsOCAssO5PH2fj2sl5glfreOBl9QF/xGVSBLqqWKhyAeTF6i2bmUOLLZ4Mqb2Yj
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4510
X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, May 10, 2022 at 02:17:34PM +0800, Lu Baolu wrote:

> +/**
> + * iommu_sva_bind_device() - Bind a process address space to a device
> + * @dev: the device
> + * @mm: the mm to bind, caller must hold a reference to mm_users
> + * @drvdata: opaque data pointer to pass to bind callback
> + *
> + * Create a bond between device and address space, allowing the device to access
> + * the mm using the returned PASID. If a bond already exists between @device and
> + * @mm, it is returned and an additional reference is taken. Caller must call
> + * iommu_sva_unbind_device() to release each reference.
> + *
> + * iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_SVA) must be called first, to
> + * initialize the required SVA features.
> + *
> + * On error, returns an ERR_PTR value.
> + */
> +struct iommu_sva *
> +iommu_sva_bind_device(struct device *dev, struct mm_struct *mm, void *drvdata)
> +{
> +	int ret = -EINVAL;
> +	struct iommu_sva *handle;
> +	struct iommu_domain *domain;
> +
> +	/*
> +	 * TODO: Remove the drvdata parameter after kernel PASID support is
> +	 * enabled for the idxd driver.
> +	 */
> +	if (drvdata)
> +		return ERR_PTR(-EOPNOTSUPP);

Why is this being left behind? Clean up the callers too please.

> +	/* Allocate mm->pasid if necessary. */
> +	ret = iommu_sva_alloc_pasid(mm, 1, (1U << dev->iommu->pasid_bits) - 1);
> +	if (ret)
> +		return ERR_PTR(ret);
> +
> +	mutex_lock(&iommu_sva_lock);
> +	/* Search for an existing bond. */
> +	handle = xa_load(&dev->iommu->sva_bonds, mm->pasid);
> +	if (handle) {
> +		refcount_inc(&handle->users);
> +		goto out_success;
> +	}

How can there be an existing bond?

dev->iommu is per-device

The device_group_immutable_singleton() insists on a single device
group

Basically 'sva_bonds' is the same thing as the group->pasid_array.

Assuming we leave room for multi-device groups this logic should just
be

	group = iommu_group_get(dev);
	if (!group)
		return -ENODEV;

	mutex_lock(&group->mutex);
	domain = xa_load(&group->pasid_array, mm->pasid);
	if (!domain || domain->type != IOMMU_DOMAIN_SVA || domain->mm != mm)
		domain = iommu_sva_alloc_domain(dev, mm);

?

And stick the refcount in the sva_domain

Also, given the current arrangement it might make sense to have a
struct iommu_domain_sva given that no driver is wrappering this in
something else.

Jason