Received-SPF: pass (google.com: domain of linux-kernel+bounces-178890-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Tue, 14 May 2024 12:55:46 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com,
	suravee.suthikulpanit@amd.com, joro@8bytes.org,
	linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
	linux-arm-kernel@lists.infradead.org, linux-tegra@vger.kernel.org,
	yi.l.liu@intel.com, eric.auger@redhat.com, vasant.hegde@amd.com,
	jon.grimm@amd.com, santosh.shukla@amd.com, Dhaval.Giani@amd.com,
	shameerali.kolothum.thodi@huawei.com
Subject: Re: [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and
 iommufd_viommu_ops
Message-ID: <ZkOJgvMNFaZZ06OO@nvidia.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <8610498e3fc00000e78bb9cef6fac9f6a54978a4.1712978212.git.nicolinc@nvidia.com>
 <ZkDMSfd9BXmsFfb+@nvidia.com>
 <ZkGKKoHnykAdGJq6@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZkGKKoHnykAdGJq6@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?Oq3Gng403JboX8igfqd6F90BdWUighjFCM+X+zvPAIR5DBFErL9Ogt2cGhGQ?=
 =?us-ascii?Q?/nbIDYIZdIFK2nyAOIXdZo7CUqPdeTHjOoxGEKoUGf4K3W0LqCGNuo77XGTw?=
 =?us-ascii?Q?VFrX/qBSuT5tvbfUAZDL2UZBBAt00jWi2T7kZUDj/pr/0nFOESCe5fYkk1dP?=
 =?us-ascii?Q?HgrieYup6iN2MpgxDLDm0TXCid/iwgkM6R3axOJ2Apfk/mhQlS59ABr7s9jL?=
 =?us-ascii?Q?/INP7o8HXEcEvCTGwnDm0qpHuYiCQm4h4PB2ss9pWbXcGoohVMGPnvJDbwjH?=
 =?us-ascii?Q?MJmu/lcE8rPYpi669onQLk28yLnCy6aWZ00g7+oWNCa26mnKM3zWX2SkqTfn?=
 =?us-ascii?Q?z7J6niQ7WuF4sF7szuXAEiMZFMYBKV/E3GdsOuB9mBVIywQbm/iDT09Sl8Gy?=
 =?us-ascii?Q?+TibPmdCw4aObvuYMWj4csZ8216JRIMHh5tx2ueaLlYljBClozqbNWPNeMAK?=
 =?us-ascii?Q?WWwXc38dvpDVgGN2CfRcqdy0RkqJOJ8TTOFnkfXyLII+q0gDgUzvhAS94MTV?=
 =?us-ascii?Q?PiK9Y/Iw2YDRQztyZ6zFKxasCKs4+JLgalx/ZtKu6VJ7EnwWWAGdmx/i7STk?=
 =?us-ascii?Q?0RPSK4iz3Qyd8SJrtrUqnsDMu5afWvRor0whCh7NQ78nh9GPy/FQQCffdaZc?=
 =?us-ascii?Q?ndQLkEy17wUjVzAmweEA5FZC4hwjXIdDOJXVz8nXVIidlX3MQtjeN6ijTQs7?=
 =?us-ascii?Q?XADkEX9YwZz/c0SVBGrK1xGGPcgHws5TnJUd/ZOTHuXqS5fFCaLTuHsiewiV?=
 =?us-ascii?Q?D+OZiCZwzZs1fyMOFJqAl53JFT8terUNLMJfqfKJRs5fDbYoS3zyurGxvSWe?=
 =?us-ascii?Q?JQEEDfGJUv85UYG3eCHYBoWvITO9VIMc75XLLtVy+jhadJ+gNJy+rHg5Imnm?=
 =?us-ascii?Q?x5VQSGYcVNS/Z0wtFbWvX7yIgQAlTNxqTyIaMgpBhtjcAaCJMCyaN5wL6cKA?=
 =?us-ascii?Q?erTJUn9P44YBoNpAPURnJScFAL2HsCsbH/hnED6nfHC9L/ZON1fhuzeqHpmZ?=
 =?us-ascii?Q?i0QYBU79qcFqy4lc8IQJn7Vgrmko6GZcL2HdiFrRllCQ6tiNJ5yGwcaxL/vM?=
 =?us-ascii?Q?iwWoLI++w+yz3H1QTcgJrecHgRPHMwwL+w7zc0ABOR90JpSPhhT/ipc03GEM?=
 =?us-ascii?Q?5KM45ePw2bMnOnLnZlHLcmxIE3PiMVaMKCGMChGPd31FihI6dr0dHsWiVHAw?=
 =?us-ascii?Q?kmfeGt5NJdlqYoWKtMP5xZq7MrpXKTe4E7oq1Pgy/pExQpkQaQ0jELSzHheY?=
 =?us-ascii?Q?01Fa2uqRdVrITDhA08JSBCYkl9AY68DqK6S4RS8f6LOUB1oUTwiTZDww2yS2?=
 =?us-ascii?Q?wRsWueGAFXxMcbkM+VjzT9fS7QAU1/uF8sqLhGZoJa2rPrRt4+OpcVNaSGA0?=
 =?us-ascii?Q?2bEhiQwHSZn+Z0+9Mam7jBoO/o/WA2lFXdhiQ+zmZe0xdEbOIN5lPyPlPOtr?=
 =?us-ascii?Q?NkQt1d41tRpILxClaQ5TNFdzZxo5XfPZPyTcUNhkCyieFf8/YFolAGc4aFay?=
 =?us-ascii?Q?7NhTG20KDUg0amiBpiN1RD6qN9ikHXOEkTzi8L77e7gh03lucPoexlzoVqqj?=
 =?us-ascii?Q?11wodZunooPN5EZNrdA=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e7f7b16f-ed07-4328-2b5d-08dc742e5230
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2024 15:55:47.7242
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QLlQeI3ajJ6EmqE4UbgRRbaJx9gQwqGWFFzjZp8mCt1pVl22SNWcCQMYY7836hN7
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6627

On Sun, May 12, 2024 at 08:34:02PM -0700, Nicolin Chen wrote:
> On Sun, May 12, 2024 at 11:03:53AM -0300, Jason Gunthorpe wrote:
> > On Fri, Apr 12, 2024 at 08:47:01PM -0700, Nicolin Chen wrote:
> > > Add a new iommufd_viommu core structure to represent a vIOMMU instance in
> > > the user space, typically backed by a HW-accelerated feature of an IOMMU,
> > > e.g. NVIDIA CMDQ-Virtualization (an ARM SMMUv3 extension) and AMD Hardware
> > > Accelerated Virtualized IOMMU (vIOMMU).
> > 
> > I expect this will also be the only way to pass in an associated KVM,
> > userspace would supply the kvm when creating the viommu.
> > 
> > The tricky bit of this flow is how to manage the S2. It is necessary
> > that the S2 be linked to the viommu:
> > 
> >  1) ARM BTM requires the VMID to be shared with KVM
> >  2) AMD and others need the S2 translation because some of the HW
> >     acceleration is done inside the guest address space
> > 
> > I haven't looked closely at AMD but presumably the VIOMMU create will
> > have to install the S2 into a DID or something?
> > 
> > So we need the S2 to exist before the VIOMMU is created, but the
> > drivers are going to need some more fixing before that will fully
> > work.
> > 
> > Does the nesting domain create need the viommu as well (in place of
> > the S2 hwpt)? That feels sort of natural.
> 
> Yes, I had a similar thought initially: each viommu is backed by
> a nested IOMMU HW, and a special HW accelerator like VCMDQ could
> be treated as an extension on top of that. It might not be very
> straightforward like the current design having vintf<->viommu and
> vcmdq <-> vqueue though...

vqueue should be considered a sub object of the viommu and hold a
refcount on the viommu object for its lifetime.
 
> In that case, we can then support viommu_cache_invalidate, which
> is quite natural for SMMUv3. Yet, I recall Kevin said that VT-d
> doesn't want or need that.

Right, Intel currently doesn't need it, but I feel like everyone will
need this eventually as the fast invalidation path is quite important.

Jason