Received-SPF: pass (google.com: domain of linux-kernel+bounces-176897-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Sun, 12 May 2024 11:03:53 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com,
	suravee.suthikulpanit@amd.com, joro@8bytes.org,
	linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
	linux-arm-kernel@lists.infradead.org, linux-tegra@vger.kernel.org,
	yi.l.liu@intel.com, eric.auger@redhat.com, vasant.hegde@amd.com,
	jon.grimm@amd.com, santosh.shukla@amd.com, Dhaval.Giani@amd.com,
	shameerali.kolothum.thodi@huawei.com
Subject: Re: [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and
 iommufd_viommu_ops
Message-ID: <ZkDMSfd9BXmsFfb+@nvidia.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <8610498e3fc00000e78bb9cef6fac9f6a54978a4.1712978212.git.nicolinc@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8610498e3fc00000e78bb9cef6fac9f6a54978a4.1712978212.git.nicolinc@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?dpyL+ruMkaj1C5O/CsHsmAqFt0cpbFVCopLskZYujbsUmKULQu19z9Dcy5WS?=
 =?us-ascii?Q?FC28x+cm7BYjv0Hv2Cp4OwdhTDP8zxU9DfnTzDJ71ST39pvaVLv1P3nQGc6x?=
 =?us-ascii?Q?ZPO85GQpz4UZK2xzLu256uJ7AVp7QY2PCpmdOYaTANrbdjLNIgXBFhK0EogG?=
 =?us-ascii?Q?69lw8F/BcvmchomVuf5qGXz/tXGF7OkEILaKW1TSkcSkicPvuLAolUE7KVe+?=
 =?us-ascii?Q?2sGxRBgGsHw00Yemwkth6/3etBiIQTW6wAuIPe9AhlF+2mEhGC0cNmCnr3iK?=
 =?us-ascii?Q?4OO++jxK8qpQi5c4NbXHFPpCPMAJDjuchpO54Yic2DiSxBg6g5MMDZhlfLDQ?=
 =?us-ascii?Q?9z2d/yqG8bAQyfAlUEyhlj1w0EB0/XwKDU1ZASHWSAq3dZnry2+OABEjsWTi?=
 =?us-ascii?Q?kA/7iSwM7n9ih2xQ5rpMkuaJybjpr/d6Hft2FEE5f2YSPLivgmSi8EeFbfAc?=
 =?us-ascii?Q?7mGCv5pWApDzCt85lFrLsN1EI+Iw1OuEAGIEBK6wohCyll347s5VprTwEtdP?=
 =?us-ascii?Q?dIYapGMnp335KVz1WsZxCcyIjwz0OWtHwccXucXjaWIdLm1Do4H89eLKmbyu?=
 =?us-ascii?Q?seMOUVy4eKyNGA5tM26jrszKYxGTHG2eJ06A/+rNR8XPWiQ8hBzN0rXAXdxB?=
 =?us-ascii?Q?HiEwEf+/7jMJJAGCg9TcsIccDGtJbVqg8pGhhK6f6wuxauQ0IiMzfWHZUHxP?=
 =?us-ascii?Q?/PsueZe8lyksCW8wXdpDEpZa/nQ1PA/B/8JjwqDbIrcYdV+eqSXnYaH4GmcP?=
 =?us-ascii?Q?RfePSv95nhGkceu2icbd1RrvA2Ie86n9++Vt15PivmKckou41q7u3FVBH3yF?=
 =?us-ascii?Q?EZnJ5bqlw4XGRtobvDkRj9qj6Lgee6j92zopqkl2n6YoDRUcG5SAD9hwA+xR?=
 =?us-ascii?Q?RN9gSZex4OstRo5jvuubQUNjWYSkU8c4kxFHwhdxZb3TxVzbJMe9fztJTBEH?=
 =?us-ascii?Q?bdiQxOzNaqvhvEeGE1sebjf1r9W1+5volVeOjaWpFTOypAq1gfewB+48AX6t?=
 =?us-ascii?Q?peeqXCw9MqI7bBXByRrHud6vVSRhXpvk8Ae0UtV0UzNBqRl8qx9goc5/PcjT?=
 =?us-ascii?Q?5DARqPv4AUgOhyAtim0dZntbnrwu+M3hx8Li+R9YrfSa+bs39dubXO1L8u2d?=
 =?us-ascii?Q?JsMOD4ICHSR1HXtkPgm9XS03PzQBwSOfvItKLsVzaqVluRJFDFjxCxepSRzw?=
 =?us-ascii?Q?bAeiDLrTcGYBHdUGFbrtnlYIircMJEYsfkTnOjXiJ8vAmuku5D1aG77VPUzD?=
 =?us-ascii?Q?s4nnbrH1/pxds90m9bBrSHdY+gB7IYyGf+T/ZKdXO+Po6bDG5GAgh8RGOr0a?=
 =?us-ascii?Q?BFV3VgPeS3cVaKudzWD8rkq4fYGYfUQ5DdLrY03hqagC+kzPkQhgbDnyTN0V?=
 =?us-ascii?Q?RhRUDWG+55X7fAj9R//03+1WIiehIVLQ08ouYU3ToNWu40jUEhMm5IPLnLwy?=
 =?us-ascii?Q?r+AeTk87X8epU2lKh4Oixrb37IonTstsO1Wv30ZGJCspadh/ZoXDI5vBB7DF?=
 =?us-ascii?Q?4BUw4Vy2Pai1ik6BwHPwpprm1HrpG0yFiAoPW/3DS/z4koUqwEfYTYCi0Wla?=
 =?us-ascii?Q?7tMDdRiKWD8SrlT1Yfp/3hrTKnYrXmlgHlMmM1Ag?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 13f5c14a-136e-4b54-41b5-08dc728fb86d
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2024 14:27:58.0602
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5VcOBtM/1n5D3Gl3H7HkUP0oGtdWWoNNp4yO+9ML+cKD+9UGBY9Sk0kKO9xjGqH1
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB8570

On Fri, Apr 12, 2024 at 08:47:01PM -0700, Nicolin Chen wrote:
> Add a new iommufd_viommu core structure to represent a vIOMMU instance in
> the user space, typically backed by a HW-accelerated feature of an IOMMU,
> e.g. NVIDIA CMDQ-Virtualization (an ARM SMMUv3 extension) and AMD Hardware
> Accelerated Virtualized IOMMU (vIOMMU).

I expect this will also be the only way to pass in an associated KVM,
userspace would supply the kvm when creating the viommu.

The tricky bit of this flow is how to manage the S2. It is necessary
that the S2 be linked to the viommu:

 1) ARM BTM requires the VMID to be shared with KVM
 2) AMD and others need the S2 translation because some of the HW
    acceleration is done inside the guest address space

I haven't looked closely at AMD but presumably the VIOMMU create will
have to install the S2 into a DID or something?

So we need the S2 to exist before the VIOMMU is created, but the
drivers are going to need some more fixing before that will fully
work.

Does the nesting domain create need the viommu as well (in place of
the S2 hwpt)? That feels sort of natural.

There is still a lot of fixing before everything can work fully, but
do we need to make some preperations here in the uapi? Like starting
to thread the S2 through it as I described?

Kevin, does Intel forsee any viommu needs on current/future Intel HW?
I assume you are thinking about invalidation queue bypass like
everyone else. I think it is an essential feature for vSVA.

> A driver should embed this core structure in its driver viommu structure
> and call the new iommufd_viommu_alloc() helper to allocate a core/driver
> structure bundle and fill its core viommu->ops:
>     struct my_driver_viommu {
>         struct iommufd_viommu core;
> 	....
>     };
> 
>     static const struct iommufd_viommu_ops my_driver_viommu_ops = {
>         .free = my_driver_viommu_free,
>     };
> 
>     struct my_driver_viommu *my_viommu =
>             iommufd_viommu_alloc(my_driver_viommu, core);

Why don't we have an ictx here anyhow? The caller has it? Just pass it
down and then it is normal:

my_viommu = iommufd_object_alloc_elm(ictx, my_viommu, IOMMUFD_OBJ_HWPT_VIOMMU, core.obj);

And abort works properly for error handling.

Jason