Received-SPF: pass (google.com: domain of linux-kernel+bounces-176896-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Sun, 12 May 2024 11:27:45 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com,
	suravee.suthikulpanit@amd.com, joro@8bytes.org,
	linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
	linux-arm-kernel@lists.infradead.org, linux-tegra@vger.kernel.org,
	yi.l.liu@intel.com, eric.auger@redhat.com, vasant.hegde@amd.com,
	jon.grimm@amd.com, santosh.shukla@amd.com, Dhaval.Giani@amd.com,
	shameerali.kolothum.thodi@huawei.com
Subject: Re: [PATCH RFCv1 05/14] iommufd: Add IOMMUFD_OBJ_VIOMMU and
 IOMMUFD_CMD_VIOMMU_ALLOC
Message-ID: <ZkDR4Rp57cy9qSqP@nvidia.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <3aa9bc1df6a2ee58a03c6ea6ededbc210a2d23a8.1712978212.git.nicolinc@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3aa9bc1df6a2ee58a03c6ea6ededbc210a2d23a8.1712978212.git.nicolinc@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?x0RoRbAi+I6U0DeEKtmJrFPf+Zo7X42yFHEYcTCCrn3fCMJH3Ys+ali/U1ot?=
 =?us-ascii?Q?yq7cXUPdZ8Frb5iZAoj9s6S9EDC1jqqmP5wydSMZWJu/WmshgYLQ5m9ko/XC?=
 =?us-ascii?Q?7gjnEa2oQF6KHczy2lt1OkuvCIdCwsCI95wWXn37yD0ycHfEq6H0hXQfjqik?=
 =?us-ascii?Q?hwhbEZek4lVNhX7ahSRlAA/nCnrOqoWivsFwaHiSPV96OxA9uouYcTAPTvBu?=
 =?us-ascii?Q?f6YED6gvvc+jpJrBazf+iPHl3O4eyGiePBmdNVi7+mAux2qXFdfmUtRJOygP?=
 =?us-ascii?Q?PeWTgGO7h1X8wJCLC6sBAaxJSSoxKdGBE3sVBP3cslVuOfttV389CNCCfirp?=
 =?us-ascii?Q?eouOtZOB8tuK4ulGYokeLWmEklpa1jP7ifyQP2i99d3S2+r1BnRBsFRgul/M?=
 =?us-ascii?Q?6tzTj2jMFQ6NZxTkF1O/qffR3VWxMv6cGpIvx7raNPQdvtyrxhjw1b9w7V+z?=
 =?us-ascii?Q?GQimCGgTKKRWa+NuFLkLqPYNyWQ+ENScOeayWEEhBJIzgWv5z6a3vYH6FZse?=
 =?us-ascii?Q?78rZC9BJSdbRsBSNEghO9bOp+ZfdEASVJ8t5CThnjpavlWFVRj+tJ5PN/TZ+?=
 =?us-ascii?Q?qx2garhVxrMVIFJdf40hgTlfZoFrYpFM9VGe6GP355aMeTKlXJl2PV6B4mV0?=
 =?us-ascii?Q?93nNAgy76Fox1x18dlJ1rg3zvuwcCkvJp/AWiXVNIykDj5de7y74dQm8XFTa?=
 =?us-ascii?Q?aYgwkB8eU/bWeQJ+052v9U0xtFBBmwb/O/CaXECMakmvezVNkTaXtkqov0nE?=
 =?us-ascii?Q?aQ5pPTjpw/V0xQ+lmQoeeSv65byb3nJT5Agttmkfp/897/PamEfe9DbX9vOG?=
 =?us-ascii?Q?6QEfWdKP/ViX8SnOfIyMYOIdadiTNZpH4JiElHP7x4kBP1WV0dweS9wO7i/h?=
 =?us-ascii?Q?5Q5ZnzOLA5NCHu9xHoakcAQkdrnLFssXHdt9xpCPATyPWb2+YPlk8XerLkfY?=
 =?us-ascii?Q?SmzJixZD6vBgrwsJ6RDafHmDmRV7u0SYcrhV/FRmZbdQxxoHXJPbNGqAo76j?=
 =?us-ascii?Q?VxIiGz4tWRivmh7kswZEm3lkGoSo41eNLYMFQR0TW2URa944XUh7C7ok6Flx?=
 =?us-ascii?Q?TZyX6aZ5ofCMJSiTWiEgzoYoFq2OxofF+GwQRUMKHZOa4brbhUuJpTUjqigd?=
 =?us-ascii?Q?TkPT4dvQcgszRaKTbhy8spNgP6GZXqmkmDKVZ3sbmA2kFsbo8+6Aip/YcHxc?=
 =?us-ascii?Q?N8tELcvmVKZbrn2/cCR6/bsjaz54EHVIRDSm7yh41e1ZCRMHLcW8epBTjXhs?=
 =?us-ascii?Q?z9Ay1VrWETPb4UJX7kyavJXhIMelCTFmM/9Iz2WocyDUBy1fLtrA8owbjx/B?=
 =?us-ascii?Q?VILp3Ta5SwwnBNr+c40gksiuoeoUV2uKfAltHljcmfi8R1GgDjXWM5kFpQlk?=
 =?us-ascii?Q?dQx2wSl3U2VKEBzX7OVmdvEcln8qcYi7M+YGJinuc01eYwQpVqtkpi973S16?=
 =?us-ascii?Q?mh1/HfIVewu3D8arhNu/hMXyP0hqidHXB618bQmuNrI4+ll/qto8TY9csXpu?=
 =?us-ascii?Q?cL4S3BCq24iC789xT2IC65Bf5dHG/RkNtflIcODVlSmLiIGsvondpVL87qWw?=
 =?us-ascii?Q?mGlzfsrQ5WDL84Y9qulHXH2oVh3mGdBWh1ppZxvG?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fc385a0e-e27d-45ec-c069-08dc728fb740
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2024 14:27:56.1727
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: v5X51SI7Ixl5cUMykSanCGqgM3OTo+qSEz+IzxhF96iaSqxKkEIyIm0rnDiyfj90
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB8570

On Fri, Apr 12, 2024 at 08:47:02PM -0700, Nicolin Chen wrote:

> +int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd)
> +{
> +	struct iommu_viommu_alloc *cmd = ucmd->cmd;
> +	struct iommufd_hwpt_paging *hwpt_paging;
> +	struct iommu_device *iommu_dev;
> +	struct iommufd_viommu *viommu;
> +	struct iommufd_device *idev;
> +	int rc;
> +
> +	if (cmd->flags)
> +		return -EOPNOTSUPP;
> +
> +	idev = iommufd_get_device(ucmd, cmd->dev_id);
> +	if (IS_ERR(idev))
> +		return PTR_ERR(idev);
> +	iommu_dev = idev->dev->iommu->iommu_dev;
> +
> +	if (!iommu_dev->ops->viommu_alloc) {
> +		rc = -EOPNOTSUPP;
> +		goto out_put_idev;
> +	}
> +
> +	hwpt_paging = iommufd_get_hwpt_paging(ucmd, cmd->hwpt_id);
> +	if (IS_ERR(hwpt_paging)) {
> +		rc = PTR_ERR(hwpt_paging);
> +		goto out_put_idev;
> +	}
> +
> +	if (!hwpt_paging->nest_parent) {
> +		rc = -EINVAL;
> +		goto out_put_hwpt;
> +	}
> +
> +	viommu = iommu_dev->ops->viommu_alloc(idev->dev, cmd->type,
> +					      hwpt_paging->common.domain);
> +	if (IS_ERR(viommu)) {
> +		rc = PTR_ERR(viommu);
> +		goto out_put_hwpt;
> +	}

Ah you did already include the S2, So should it be
domain->viommu_alloc() then?

> +
> +	/* iommufd_object_finalize will store the viommu->obj.id */
> +	rc = xa_alloc(&ucmd->ictx->objects, &viommu->obj.id, XA_ZERO_ENTRY,
> +		      xa_limit_31b, GFP_KERNEL_ACCOUNT);
> +	if (rc)
> +		goto out_free;
> +
> +	viommu->obj.type = IOMMUFD_OBJ_VIOMMU;

See my other notes, lets try not to open code this.

> +	viommu->type = cmd->type;
> +
> +	viommu->ictx = ucmd->ictx;
> +	viommu->hwpt = hwpt_paging;
> +	viommu->iommu_dev = idev->dev->iommu->iommu_dev;
> +	cmd->out_viommu_id = viommu->obj.id;
> +	rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
> +	if (rc)
> +		goto out_erase_xa;
> +	iommufd_object_finalize(ucmd->ictx, &viommu->obj);
> +	refcount_inc(&viommu->hwpt->common.obj.users);
> +	goto out_put_hwpt;
> +
> +out_erase_xa:
> +	xa_erase(&ucmd->ictx->objects, viommu->obj.id);
> +out_free:
> +	if (viommu->ops && viommu->ops->free)
> +		viommu->ops->free(viommu);
> +	kfree(viommu);

This really should use the abort flow. The driver free callback has to
be in the object release..

> +
> +/**
> + * enum iommu_viommu_type - VIOMMU Type
> + * @IOMMU_VIOMMU_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension for SMMUv3
> + */
> +enum iommu_viommu_type {
> +	IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
> +};

At least the 241 line should be in a following patch

> +/**
> + * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
> + * @size: sizeof(struct iommu_viommu_alloc)
> + * @flags: Must be 0
> + * @type: Type of the VIOMMU object. Must be defined in enum iommu_viommu_type
> + * @dev_id: The device to allocate this virtual IOMMU for
> + * @hwpt_id: ID of a nested parent HWPT
> + * @out_viommu_id: Output virtual IOMMU ID for the allocated object
> + *
> + * Allocate an virtual IOMMU object that holds a (shared) nested parent HWPT
> + */
> +struct iommu_viommu_alloc {
> +	__u32 size;
> +	__u32 flags;
> +	__u32 type;
> +	__u32 dev_id;
> +	__u32 hwpt_id;
> +	__u32 out_viommu_id;
> +};

This seems fine.

Let's have a following patch to change the hwpt_alloc to accept the
viommu as a hwpt as a uAPI change as well. 

The more I think about how this needs to work the more sure I am that
we need to do that.

ARM will need a fairly tricky set of things to manage the VMID
lifecycle. In BTM mode the VMID must come from the KVM. For vcmdq the
VMID is needed to create the queue/viommu. For AMD the S2 is needed to
create the VIOMMU in the first place.

So, to make this all work perfectly we need approx the following
 - S2 sharing across instances in ARM - meaning the VMID is allocated
   at attach not domain alloc
 - S2 hwpt is refcounted by the VIOMMU in the iommufd layer
 - VIOMMU is refcounted by every nesting child in the iommufd layer
 - The nesting child holds a pointer to both the S2 and the VIOMMU
   (viommu optional)
 - When the nesting child attaches to a device the STE will source the
   VMID from the VIOMMU if present otherwise from the S2
 - "RID" attach (ie naked S2) will have to be done with a Nesting
   Child using a vSTE that indicates Identity. Then the attach logic
   will have enough information to get the VMID from the VIOMMU
 - In full VIOMMU mode the S2 will never get a VMID of its own, it
   will always use the VIOMMU. Life cycle is simple, the VMID is freed
   when the VIOMMU is freed. That can't happen until all Nesting
   Children are freed. That can't happen until all Nesting Children
   are detached from devices. Detatching removes the HW touch of the VMID.

At this point you don't need the full generality, but let's please get
ready and get the viommu pointer available in all the right spots and
we can keep the current logic to borrow the VMID from the S2 for the
VIOMMU.

AMD folks, please consider if this works for you as well.

Jason