Received-SPF: pass (google.com: domain of linux-kernel+bounces-188387-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Nicolin Chen <nicolinc@nvidia.com>, "jgg@nvidia.com" <jgg@nvidia.com>
CC: "will@kernel.org" <will@kernel.org>, "robin.murphy@arm.com"
	<robin.murphy@arm.com>, "suravee.suthikulpanit@amd.com"
	<suravee.suthikulpanit@amd.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Thread-Topic: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Thread-Index: AQHajVVl26WkVBqVE0qDV+tml8yTY7Gkmr6AgAGEQoCAABbGEA==
Date: Fri, 24 May 2024 07:13:23 +0000
Message-ID: <BN9PR11MB52762A0BC67B64D4171480C78CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <c97a98a72ee3498c587e5898d6b899553ccd9b27.1712978212.git.nicolinc@nvidia.com>
 <BN9PR11MB5276A897A5386DFAC9A35F9D8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAoN3s96HL7ROth@nvidia.com>
In-Reply-To: <ZlAoN3s96HL7ROth@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?hgJw9xeNZFfpJou7CRYYF2E9PZgofVzJ59DCY8Wwvn13SgzzpyxpfGNRNy1w?=
 =?us-ascii?Q?cGsd2/TdadiMLBuRBPwygBdJNu+UxebJFk+YVgWd30Fozmu3FAQi9lHalM7E?=
 =?us-ascii?Q?TQJ8dmzqkqgUcm9Ucal7YYkjGG2kUJSUbgePaNgEj6oMux9wJRMmslZBvYTl?=
 =?us-ascii?Q?KMvWkzxwlofQ6eosIe0ogZ/G2O28SnPpJ6F77aQKpmPK5/wyNaHyhrDF7Emo?=
 =?us-ascii?Q?egqrlcvgedax0qO8zje+grEkOBXq5PktNBya2WOLgGuO/RgTAPYJ8T7cP1+f?=
 =?us-ascii?Q?FwPPS2jzOO4eg5wEnR8jwhW6wPvoSQS3hoxMN/uCj5xzpf4jON2pzWp3Vp7k?=
 =?us-ascii?Q?3H2o+YvySjglv6X/JIZvxR1J5YY4LfxdYJzTe1w84GyykQoGsZkqvl5u2UKh?=
 =?us-ascii?Q?S3ipO0IRb2NAOkBo8K9fDCka3M51fR+OGxg5Qwxi3LR5quPwjU84pcUmxbef?=
 =?us-ascii?Q?xHvvnL0WVeHzhvmK/YYqjX7xQL82MNapuCTxmZ6IMKPXJ5a6l8PNoojpVD/d?=
 =?us-ascii?Q?E1DXZWiuStlZ8Y4lJRFLU31JC+QLB5V5Z/S8rQXpyPHsAM00C3Vsq4Mci2Rk?=
 =?us-ascii?Q?hsTnTzjMEenJu2YUepN+CYSgFAEL13tNCZshra1aAMNx7uHGUIKXweifCvmp?=
 =?us-ascii?Q?lEQcR4NCAZxJAw5RFZo6/Zkhmmd5HaYXGlNJCVwLUKd1gXM/MB5bu0+arMGV?=
 =?us-ascii?Q?2aOpJmGk7xqXR4eid17AJArBNT6lB/bypGFe4oGIrS22Dkh16gGAQxALc3aR?=
 =?us-ascii?Q?jZqZOFTBghKQ2KHcuSi5vOrSwNiyP8uhme0nM/YartD/SBNbG0n4BvTXIGv1?=
 =?us-ascii?Q?nF91GEQoFMhtvZozhnAvhNkAVolOupDjrJxGUH3rNlxq/HuyvtFFVtO1oKmI?=
 =?us-ascii?Q?rJH/uc2CHp7i3/TLTNWV0n/JwJ6+olG0xahJ0EjDyLLoGLDp3xn+BMG8iYKz?=
 =?us-ascii?Q?KIvwjThRMq7jobcxoa98afc0DRDns9QsyUqQ5j+HpgDlpbZjHfMaXfLKLIIh?=
 =?us-ascii?Q?9ON+lNSzC8zloBuvqYtYSsyzNeV8hZUGCRlNqA9YL/7G9ds6/tCcGIeZxo8l?=
 =?us-ascii?Q?Ttg7TpYiVpr1D5NbIH92z3B67zjarCfxu05yG2j37rrzlwOkW6uvIIkGbah2?=
 =?us-ascii?Q?H2yqlkkqN6OBQ8oKjS2Trj4T64Q/bND/aIO2YHn3DXGeCdmNAsZApV7GEfeN?=
 =?us-ascii?Q?xHqx7v87OgsH293gx4e/N1s6AL9wD7L2cQkjz8c8L/09BSYDeP18jjQzrWWZ?=
 =?us-ascii?Q?2foWZD3qKbvoAGY1S+TKM7ILy1WzvneTAMgtQHYQ1GOGx8AQ8yTFxb3IUWhI?=
 =?us-ascii?Q?6Fdl/5quws5vPNTjH2xp0qw8nDVOTQzQbjEt5vvr0DVFjL0wqbamr8XKYSnw?=
 =?us-ascii?Q?n0QiPxmlvYQYgWWhKHacTR+uBsWA4BfG3yC1EdBld/QPGVpLRoC++aC5ZdfY?=
 =?us-ascii?Q?8d9XkeAb1NegqMnGEsp1TNCWd5DAcR67cuIHGaSwFCTzcrZ2R8gPoc1AZmWC?=
 =?us-ascii?Q?6rds1fLHR9ZQgJeKrqVlbFfPvdmE5Dw8LnSU/XTBkKPxkBfUQw1mWESew+xM?=
 =?us-ascii?Q?td4hpCJL+3pd6zCmHoIwbcIuDIwsW5yVHYy4qJHP?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5634cbfe-bd2f-45bb-1191-08dc7bc0ffd1
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 May 2024 07:13:23.5194
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 43Ncx6yk2M1jsVB8yhRiQ++6txvqjFBaGOKxXDMD5caQZDmO1zKX/1yuwxpO2KEnUzCl1MfRkov2OxUDuDzPGw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8244
X-OriginatorOrg: intel.com

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, May 24, 2024 1:40 PM
>=20
> On Thu, May 23, 2024 at 06:42:56AM +0000, Tian, Kevin wrote:
> > btw there is a check in the following code:
> >
> > +       if (viommu->iommu_dev !=3D idev->dev->iommu->iommu_dev) {
> > +               rc =3D -EINVAL;
> > +               goto out_put_viommu;
> > +       }
> >
> > I vaguely remember an earlier discussion about multiple vSMMU instances
> > following the physical SMMU topology, but don't quite recall the exact
> > reason.
> >
> > What is the actual technical obstacle prohibiting one to put multiple
> > VCMDQ's from different SMMU's into one vIOMMU instance?
>=20
> Because a VCMDQ passthrough involves a direct mmap of a HW MMIO
> page to the guest-level MMIO region. The MMIO page provides the
> read/write of queue's head and tail indexes.
>=20
> With a single pSMMU and a single vSMMU, it's simply 1:1 mapping.
>=20
> With a multi-pSMMU and a single vSMMU, the single vSMMU will see
> one guest-level MMIO region backed by multiple physical pages.
> Since we are talking about MMIO, technically it cannot select the
> corresponding MMIO page to the device, not to mention we don't
> really want VMM to involve, i.e. no VM exist, when using VCMDQ.

can a vSMMU report to support multiple CMDQ's then there are
several MMIO regions each mapped to a different backend VCMDQ?

but I guess even if it's supported there is still a problem describing
the association between assigned devices and the CMDQ's of the
single vIOMMU instance. On bare metal a feature of multiple CMDQ
is more for load-balancing so there won't be a fixed association
between CMDQ/device. But the fixed association exists if we mixes
multiple VCMDQ's in a single vIOMMU instance then there may
lack of a way in spec to describe it.

>=20
> So, there must be some kind of multi-instanced carriers to hold
> those MMIO pages, by attaching devices behind different pSMMUs to
> corresponding carriers. And today we have VIOMMU as the carrier.
>=20
> One step back, even without VCMDQ feature, a multi-pSMMU setup
> will have multiple viommus (with our latest design) being added
> to a viommu list of a single vSMMU's. Yet, vSMMU in this case
> always traps regular SMMU CMDQ, so it can do viommu selection
> or even broadcast (if it has to).
>=20

I'm curious to learn the real reason of that design. Is it because you
want to do certain load-balance between viommu's or due to other
reasons in the kernel smmuv3 driver which e.g. cannot support a
viommu spanning multiple pSMMU?