Received-SPF: pass (google.com: domain of linux-kernel+bounces-185960-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>, Nicolin Chen <nicolinc@nvidia.com>
CC: "will@kernel.org" <will@kernel.org>, "robin.murphy@arm.com"
	<robin.murphy@arm.com>, "suravee.suthikulpanit@amd.com"
	<suravee.suthikulpanit@amd.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and
 iommufd_viommu_ops
Thread-Topic: [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and
 iommufd_viommu_ops
Thread-Index: AQHajVVjs28Dvjq/i0qw8WNB6ldTtbGTz8aAgADiWwCAAmGRAIAMGHog
Date: Wed, 22 May 2024 08:58:34 +0000
Message-ID: <BN9PR11MB5276423A0BFBDA8346E1ED3C8CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <8610498e3fc00000e78bb9cef6fac9f6a54978a4.1712978212.git.nicolinc@nvidia.com>
 <ZkDMSfd9BXmsFfb+@nvidia.com> <ZkGKKoHnykAdGJq6@nvidia.com>
 <ZkOJgvMNFaZZ06OO@nvidia.com>
In-Reply-To: <ZkOJgvMNFaZZ06OO@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?iZrT61nwssRXoGHJgnouyDye66kSk31CF72VSAxkhS+MLyfCzgnTv3rcRFMS?=
 =?us-ascii?Q?s3ilrivrN6mzd15Pju/vBett4yt7WNMiGXbbUK7gBys3Kx5w1HLBB5lr0tcE?=
 =?us-ascii?Q?kq39WfbtTp7uMQu0pfcdEG4yYmIwTwqciwgxSQVBKPfFVYPwPvS8Y6wHeft/?=
 =?us-ascii?Q?QDVuE0qxSvE1uB/E4P7XORvH2cCcC2Sqy/2j/VPQDsewPB3YHBRO4ZhNTTQB?=
 =?us-ascii?Q?WLvy0sN+KKD9oynW9S7aswDgKPNQ3c/HQkOkLp9uG220KLuvugVXBTbV0FAT?=
 =?us-ascii?Q?zaZMYGNJmZLwS1OGoMDOSJdaTWxwzTn41XGUOTyXW9oskM/mueWXwU6P7Q4m?=
 =?us-ascii?Q?kgjaEWgHBcw9QwdHCx78ZRQi8zyXLobX/ONETRF1sOT6D9pSppPDZ/6mbZJR?=
 =?us-ascii?Q?CHsQ5g4GVtGCNVCnhiba/5mnR/9RLs8lDoR5I3y/qe2LaHj9GN1St5L9epd0?=
 =?us-ascii?Q?jnegLL5rdC8w+eulhjLow64TiJQXalZ0SV+aFJDJ1CE6/Je3u9BkNCKVEO94?=
 =?us-ascii?Q?J1KC/Ky1Hksvy9WE8caCjh471NrBLMGIVNMNHDIkjFxMHM7Gw78nLg2vlzZo?=
 =?us-ascii?Q?Gm15rx3oFswu/i1aw9k/C2BP7qTxJ/ABMWD9EmgUJIwHRuKbhrOhxAhCZzhL?=
 =?us-ascii?Q?FsKFd5YAMZxEk1NOg8k97z9g1UpGlcmsd3HqVtSiz3Ggifox/re5jGq4e8Sj?=
 =?us-ascii?Q?wMVSi7lQ96yKyu119lRi8D8HqyMWszrASt94rJgYFOWHhsv/5WsZcPX05oyJ?=
 =?us-ascii?Q?tA4B1VIch+bHmsw1KgUzjMzAdVmuMI2T3AIj8JaPTtMAHMXYcy7JOKuneNBy?=
 =?us-ascii?Q?V32Y7bWn6H1xmTm/hN8Qui0Qxd3kKSj77JmcTXCKWB6CuaGBNuPT+4vHmmfP?=
 =?us-ascii?Q?ZWSyaLlibeAbjtcv4KhlB04idsPXSlyVVJNdljJUVtCOnLkOYgKgsEOjohXU?=
 =?us-ascii?Q?zzlKlEORA3tLcGh16o4mr/Gmh8lwGYasOevSMF7khdzcHFyby9kwwmX33xGd?=
 =?us-ascii?Q?FK8u27yw2Tc/OsVe68awumWp3Xbo4ATzDMYAekm/pv1ON6czUBe//byebRc+?=
 =?us-ascii?Q?V1bOhuBiYx/NAWUy9VKGgXqHrf0UF3FEc/dw4rXsXb8IvYjzyVl8nHfw9h6X?=
 =?us-ascii?Q?pFNOA2R0FogeO4sFJPOVOmKXkY/7fmxwNKhzsCv4urSRpDL9WqkX3VTLrZPV?=
 =?us-ascii?Q?kkYiL8y7SMIUJv1Ni/02k6QM5xkFtK8v1cODl4/OJMH30t54+H7GES0JCCfJ?=
 =?us-ascii?Q?xu0BZxF7EQgI7S3Nur06luziMCkLk8hbGz3Odxekbs2gorR+CO+xJ2BYnWp4?=
 =?us-ascii?Q?zSipbbeGxqQ9slLBf/fqap6ari+uIRvL5kNndzmIdGe5f2LanbycAled+M0t?=
 =?us-ascii?Q?xmZtewp7gfTbN4Px4biUs+GmihE6vRLitFhQj8NAtQQ3U4GwumyRE5Ujk3Ob?=
 =?us-ascii?Q?nV3VBWkgoJqtwKo1h3EgALyu6dJHnlrlzXgqeyWAacr5VyzUPZEueZ8wLhgZ?=
 =?us-ascii?Q?NeYWWz0MFqXeSVxmriTj3fn79bUPxHyaJInLd60ASI/NZvPskj1vNhYTTNh/?=
 =?us-ascii?Q?YWk3Wj+C2neCeUXqDWP98AEs6rgUrxqitg3rFLnm?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ea0ee4d9-a676-42b9-1d88-08dc7a3d5c89
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 May 2024 08:58:34.3420
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 87waCpnzHFjOig8p5hm/SuubPmRDxImrr6d71VEYzYMsJ8XS7ZI5OhhI9MRFuLGVLjvdr73YZKxfVAV9grCApw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7849
X-OriginatorOrg: intel.com

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, May 14, 2024 11:56 PM
>=20
> On Sun, May 12, 2024 at 08:34:02PM -0700, Nicolin Chen wrote:
> > On Sun, May 12, 2024 at 11:03:53AM -0300, Jason Gunthorpe wrote:
> > > On Fri, Apr 12, 2024 at 08:47:01PM -0700, Nicolin Chen wrote:
> > > > Add a new iommufd_viommu core structure to represent a vIOMMU
> instance in
> > > > the user space, typically backed by a HW-accelerated feature of an
> IOMMU,
> > > > e.g. NVIDIA CMDQ-Virtualization (an ARM SMMUv3 extension) and
> AMD Hardware
> > > > Accelerated Virtualized IOMMU (vIOMMU).
> > >
> > > I expect this will also be the only way to pass in an associated KVM,
> > > userspace would supply the kvm when creating the viommu.
> > >
> > > The tricky bit of this flow is how to manage the S2. It is necessary
> > > that the S2 be linked to the viommu:
> > >
> > >  1) ARM BTM requires the VMID to be shared with KVM
> > >  2) AMD and others need the S2 translation because some of the HW
> > >     acceleration is done inside the guest address space
> > >
> > > I haven't looked closely at AMD but presumably the VIOMMU create will
> > > have to install the S2 into a DID or something?
> > >
> > > So we need the S2 to exist before the VIOMMU is created, but the
> > > drivers are going to need some more fixing before that will fully
> > > work.

Can you elaborate on this point? VIOMMU is a dummy container when
it's created and the association to S2 comes relevant only until when
VQUEUE is created inside and linked to a device? then there should be
a window in between allowing the userspace to configure S2.

Not saying against setting S2 up before vIOMMU creation. Just want
to better understand the rationale here.

> > >
> > > Does the nesting domain create need the viommu as well (in place of
> > > the S2 hwpt)? That feels sort of natural.
> >
> > Yes, I had a similar thought initially: each viommu is backed by
> > a nested IOMMU HW, and a special HW accelerator like VCMDQ could
> > be treated as an extension on top of that. It might not be very
> > straightforward like the current design having vintf<->viommu and
> > vcmdq <-> vqueue though...
>=20
> vqueue should be considered a sub object of the viommu and hold a
> refcount on the viommu object for its lifetime.
>=20
> > In that case, we can then support viommu_cache_invalidate, which
> > is quite natural for SMMUv3. Yet, I recall Kevin said that VT-d
> > doesn't want or need that.
>=20
> Right, Intel currently doesn't need it, but I feel like everyone will
> need this eventually as the fast invalidation path is quite important.
>=20

yes, there is no need but I don't see any harm of preparing for such
extension on VT-d. Logically it's clearer, e.g. if we decide to move
device TLB invalidation to a separate uAPI then vIOMMU is certainly
a clearer object to carry it. and hardware extensions really looks like
optimization on software implementations.

and we do need make a decision now, given if we make vIOMMU as
a generic object for all vendors it may have potential impact on
the user page fault support which Baolu is working on. the so-called
fault object will be contained in vIOMMU, which is software managed
on VT-d/SMMU but passed through on AMD. And probably we don't
need another handle mechanism in the attach path, suppose the
vIOMMU object already contains necessary information to find out
iommufd_object for a reported fault.

Baolu, your thoughts?