Received-SPF: pass (google.com: domain of linux-kernel+bounces-186833-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>, Nicolin Chen <nicolinc@nvidia.com>
CC: "will@kernel.org" <will@kernel.org>, "robin.murphy@arm.com"
	<robin.murphy@arm.com>, "suravee.suthikulpanit@amd.com"
	<suravee.suthikulpanit@amd.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH RFCv1 00/14] Add Tegra241 (Grace) CMDQV Support (part 2/2)
Thread-Topic: [PATCH RFCv1 00/14] Add Tegra241 (Grace) CMDQV Support (part
 2/2)
Thread-Index: AQHajVVgzSKGnSPg+UW1Z+6EC5j/q7GjLESAgACIwgCAADIFgIAAPc+AgAADkSA=
Date: Wed, 22 May 2024 23:43:51 +0000
Message-ID: <BN9PR11MB527613C6094EDE9B2732928F8CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <BN9PR11MB527641C15DD88FB0C44323D08CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522164818.GB20229@nvidia.com> <Zk5Lx7IqvnE2q7a7@nvidia.com>
 <20240522232833.GH20229@nvidia.com>
In-Reply-To: <20240522232833.GH20229@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?iqPh/llYkh0X33q69qD/jGagbgC2/Xnl49FUeVYnJdT9oa1uZdxgAhtKh7zN?=
 =?us-ascii?Q?QRswdBUsSP6fKjZayHiezaKyFJWUdFmFYpurPToCEpYvOfW6/VCbDUdd6ZCu?=
 =?us-ascii?Q?JxNO7wAuI6gKxwloEjpiaBztbKuCMkqqmTCDjq5pd3dhj/68KfZIkrjon/vk?=
 =?us-ascii?Q?Cg+rHewX4C9faiN5xbG7xFSWk5tIlre6R5eeuQCrqFsg8IY3kUlZnerL8xoA?=
 =?us-ascii?Q?fVuGlMPNojMX1JtT6bPy6oNl4Y2BWmumj7PLlUI4bIz39ev0k1aCd1ZcwShZ?=
 =?us-ascii?Q?r51G26MscimlNXMCJCyyKxrpSCdPnEy23MUTybxAYxxYY/WnxqVT4T4yNr2K?=
 =?us-ascii?Q?cFRWVK830GqdQBsyRHicUTugYKi+vYeold/iVP2WAABCLjdwl6YTljXcTH4c?=
 =?us-ascii?Q?Jweds4D+RkKVSQGolrav74dEW6ECXtfDFiHtayktEfJfwgP+XuQ5s3nP/Ncv?=
 =?us-ascii?Q?dOfvb0mQuhEHZ57gwblRPocqYI72cYLgmCkxMiCiobFQb0HsOUItYfPQczgl?=
 =?us-ascii?Q?amgDTXAgtYDBvfHgP7s4MdROAPuXC9rVLK1AsGnKIOuNh3MBmkB5SWzyt7C0?=
 =?us-ascii?Q?vgDS3qFk1Dshm5Smk3pIUlo6Bv72WVK9RrSRjczf5MEmw1ZPe7INW2+KOKGs?=
 =?us-ascii?Q?/sgDe1EZvHkhj0j/uaaqPXmZgGurre0r5ZLfuzTpmLbtEMrK4F8hcBeYu4hQ?=
 =?us-ascii?Q?vugb0bTbiUsuVidfLqe1uuB04VzuSRYHRhn1w4poJwjYNr4lP6rNTcW8ghje?=
 =?us-ascii?Q?2mWErYD2c4cKzYYze2nsBv2p7AQ1NAPZef3PcjznDKOhLxM+t5B2ad8SW4Wk?=
 =?us-ascii?Q?zJdw4XteugXYEydPkq8N3EoFxI6S1xbDQ7Hc9HEBE7xLjATMDQ7lV2B43U89?=
 =?us-ascii?Q?xGMI/UjCzsnr4uIvqNeJ5x8J/ABBcyTJziKktIBUYkf1hqPFFZvKm+sw3rfy?=
 =?us-ascii?Q?PZ4cx8eKswrjzuKjpDxr5gPnhCjY15YLxynsMs4Xu4tg4LAQLX2d6n/pco3s?=
 =?us-ascii?Q?THB3LAAa72yLmd0mfZ4w3S4r0KGgaPwVRAk7/f8SrPSne9o02C6HdCCxSFfr?=
 =?us-ascii?Q?SkcieDgzDFTuDiRrGmXJNAyiOtkWxasUxkhper0pw2/Yr1Q7iNE1CL/j/yLY?=
 =?us-ascii?Q?gFSZvhSpkQ7qbKhyVtpPzBn257+9AK4rNkiqh+EElR/mmhsP1bEcgAAZFdTy?=
 =?us-ascii?Q?1H5eRodVuo87zs39ybF1c6aQSTVjqR3RlRR1VMWmgJSmZd5iaM/d45+qRgHW?=
 =?us-ascii?Q?iAdhMDkuINShJpZ848pk1l9DG4IenFtWbRBM7ZKWG9MSApviTtFqIIbNcCiQ?=
 =?us-ascii?Q?oyky9vhfrPpvZGqARQ/XFrD8pRF7zwPgxHcBN+zlNQR7LbCChWoFnkAGQFnP?=
 =?us-ascii?Q?8uEN1Ugy+Cyan0JZwRafWo6V0Vz0+9Bi2u6oOmL1Tist2tp3aLNh/eQFTFxf?=
 =?us-ascii?Q?re8XAa/VeW3lQzpwDvk5pI66xocsbHfu2xx3rFLGTx+ywP6palcmWuKsSXe0?=
 =?us-ascii?Q?Pj6BwRwY+HTqduOGJEpOY4zhVuoEDyUhU71GXfY0XVON3Wps8REkYUEAc8Lp?=
 =?us-ascii?Q?sLn0ZqLHmns50NdZK9eN4ayUbHH2jELRYuYBc93a?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5a98653f-a569-4d6e-6d24-08dc7ab90905
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 May 2024 23:43:51.8144
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 9B62rw5v0OzlExL/41uNvhgRSO4b9e3GdasrAK/DBLgFHWlafo2DqW1C8M/24SdCnWb85/F6fjYtL6RYoIX7Fg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR11MB8727
X-OriginatorOrg: intel.com

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, May 23, 2024 7:29 AM
>=20
> On Wed, May 22, 2024 at 12:47:19PM -0700, Nicolin Chen wrote:
> > On Wed, May 22, 2024 at 01:48:18PM -0300, Jason Gunthorpe wrote:
> > > On Wed, May 22, 2024 at 08:40:00AM +0000, Tian, Kevin wrote:
> > > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > > > Sent: Saturday, April 13, 2024 11:47 AM
> > > > >
> > > > > This is an experimental RFC series for VIOMMU infrastructure, usi=
ng
> NVIDIA
> > > > > Tegra241 (Grace) CMDQV as a test instance.
> > > > >
> > > > > VIOMMU obj is used to represent a virtual interface (iommu) backe=
d
> by an
> > > > > underlying IOMMU's HW-accelerated feature for virtualizaion: for
> example,
> > > > > NVIDIA's VINTF (v-interface for CMDQV) and AMD"s vIOMMU.
> > > > >
> > > > > VQUEUE obj is used to represent a virtual command queue (buffer)
> backed
> > > > > by
> > > > > an underlying IOMMU command queue to passthrough for VMs to
> use
> > > > > directly:
> > > > > for example, NVIDIA's Virtual Command Queue and AMD's Command
> Buffer.
> > > > >
> > > >
> > > > is VCMDQ more accurate? AMD also supports fault queue passthrough
> > > > then VQUEUE sounds broader than a cmd queue...
> > >
> > > Is there a reason VQUEUE couldn't handle the fault/etc queues too? Th=
e
> > > only difference is direction, there is still a doorbell/etc.

No reason. the description made it specific to a cmd queue which
led me the impression that we may want to create a separate
fault queue.

> >
> > Yea, SMMU also has Event Queue and PRI queue. Though I haven't
> > got time to sit down to look at Baolu's work closely, the uAPI
> > seems to be a unified one for all IOMMUs. And though I have no
> > intention to be against that design, yet maybe there could be
> > an alternative in a somewhat HW specific language as we do for
> > invalidation? Or not worth it?
>=20
> I was thinking not worth it, I expect a gain here is to do as AMD has
> done and make the HW dma the queues directly to guest memory.
>=20
> IMHO the primary issue with the queues is DOS, as having any shared
> queue across VMs is dangerous in that way. Allowing each VIOMMU to
> have its own private queue and own flow control helps with that.
>=20

and also shorter delivering path with less data copy?