Received-SPF: pass (google.com: domain of linux-kernel+bounces-193387-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Nicolin Chen <nicolinc@nvidia.com>
CC: Jason Gunthorpe <jgg@nvidia.com>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Thread-Topic: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Thread-Index: AQHajVVl26WkVBqVE0qDV+tml8yTY7Gkmr6AgAGEQoCAABbGEIAAaX4AgAPpupCAAtXxAIAAbUsA
Date: Wed, 29 May 2024 02:58:11 +0000
Message-ID: <BN9PR11MB52762EA9B444DA71F551C3868CF22@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <c97a98a72ee3498c587e5898d6b899553ccd9b27.1712978212.git.nicolinc@nvidia.com>
 <BN9PR11MB5276A897A5386DFAC9A35F9D8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAoN3s96HL7ROth@nvidia.com>
 <BN9PR11MB52762A0BC67B64D4171480C78CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240524131912.GT20229@nvidia.com>
 <BN9PR11MB5276BBD592021507C3A0EBB38CF02@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlY886FT3HHLC8Cf@nvidia.com>
In-Reply-To: <ZlY886FT3HHLC8Cf@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RVYm1IehQnzZRF1Yijw1StnFS72sZ0/Y7QSUACCjdkTtf98Dw2meoXnIRcUn?=
 =?us-ascii?Q?hgjgsS4WYsWMVtimqlL388amhIRGm1m6L2Lt6KiXnTMQ3bgWhpNuVkSlQPKv?=
 =?us-ascii?Q?cLlnOCCQTgvJJIIKOvRDRbYz5UPoUqhp2DyPscnWv4AWiQ5qq4qGd2TRzvRZ?=
 =?us-ascii?Q?5nUXVvOoDc7R/zCEiCT5fq38imLjWVmRMP1qvaPzIowGmuMLHpQ2N1ORRECa?=
 =?us-ascii?Q?AuAMrwLPkWST/CU28RLtLkx2k/deL7DTIuIwXjQG1F2lTgaKVpzgaK9IHhgW?=
 =?us-ascii?Q?CfeAYtDOpM4tzmf9xWDgtVnLv5mffCtKf4gEf1EsYdMAo/QuH43tx4hDbxxu?=
 =?us-ascii?Q?UWAic1O4uCCWjSR2xAq4889rrtyIHhnDI9uEe5VSrML4AulN8LJ3hS6OMlIL?=
 =?us-ascii?Q?zwR5Bzh6WdIZZFj0JhDvgazVaOGrVLX8X2cPT0tBi6YIO2Q2vR1AkKt9162u?=
 =?us-ascii?Q?xK0ui1NdzQJF8SBWxI/DvOUUNZVwkJ+Njf7R7umSNIAapLfT9TOG5+V0nTfb?=
 =?us-ascii?Q?4biKEkZ0cK2+236ZzaBpJ09CvlGxG925Wwdwlhnuxz8AihSUZYI5kD42/hUc?=
 =?us-ascii?Q?G/YeOI0VHdjMCsRwnXp4ehegmE6nL/yOVkkIeKecyM/E0iOYODJ4qKhAkY25?=
 =?us-ascii?Q?mwTdVjJeelrwJqdWOeZhdF0X4QSk0QuDzIWv4Gd9/DoSR7MTBGizIMW6tKYl?=
 =?us-ascii?Q?s0dc+rs9bT9H7Iux51o3bmyfK8gYyapM+NMaRe5bx/O80cT3g33423IKWfHW?=
 =?us-ascii?Q?qOcKdWv5IqAEjjcHDQMHqFTLE4P7R5cQG+U0KZcENScurzDjOoY7PbScgQUm?=
 =?us-ascii?Q?JpqUtrwDc+cRYreSAw3e8icb44O4YrerB1ttzGii47zwR9WEcZfCWmWxaIMR?=
 =?us-ascii?Q?E/iNk7wd9+2MfzMW2PNOnfgXUJDJ0GwuT7O97TBSkBxy+cxmPKqa1vlUBzth?=
 =?us-ascii?Q?gSFBrOHGwGsrfXm30AwNBbO9ei82cjYRv8JGmJ65xEKzCmEsNwloQYCs4ULr?=
 =?us-ascii?Q?ubHf772JqoVgcvTS7MSMYEnmkRbvVPOslOyeSbUuy5dXiAxZ0A7Uf9/9OqjM?=
 =?us-ascii?Q?sskZzYT9PpfSWa8jtnkN1ugzaZrj5GNnXGR1nRcAeWvdenzrnSP2D038pzKk?=
 =?us-ascii?Q?S2BF139OHAX884hRHucdAnHm43U4IbuAiI6scaUDEXHsJnrf0EGfZghtf/Lb?=
 =?us-ascii?Q?DzfupMGBmGwbZsOPbeqwyctgdHIvO2c4KUFv4Jb4O3QHEFlL9qT1v9A4aHub?=
 =?us-ascii?Q?JebvAc0RamdoMIH0pMjwFNcfKFQ/VDufLs9QvYYgOdY8Vy/ynwxK2BKB3De5?=
 =?us-ascii?Q?+MWkineR7pNiOZAbIboOUUvJI/S/bnp30y1h04opZVSO/JplM9N2XTT6rUpv?=
 =?us-ascii?Q?v5BtmZKQc4DIIrdSt+419DbSmrmjTwUvQj807zC99guGCKDGHKAQP02RWeAq?=
 =?us-ascii?Q?kV6Sz3QXj0TfXKh82cC3hG3Ll+ZDpvQJi+ZH9gMpd8ObIMnB5sxgSW0BXtNF?=
 =?us-ascii?Q?eeMAUlr0J5+FBGFKI7uy6L6uD8PWsxQFeQ9S9KTYNcU2X4YD1uY2RokYom0A?=
 =?us-ascii?Q?NI9dM/GdP3+X1KbzGPpp7J3KBuyMBjxle62/aIe0?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 027302bb-9e78-4b7a-ff15-08dc7f8b2d3f
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 May 2024 02:58:11.6001
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: SYSW5gVTcnWSiBmY+a91AxCy4fK904Z0QD0sQQy1qplnoESOhyzP+7ywtZYJfC/wBi1/0HeDUFc90L0r6q2AqQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB5163
X-OriginatorOrg: intel.com

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Wednesday, May 29, 2024 4:23 AM
>=20
> On Mon, May 27, 2024 at 01:08:43AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Friday, May 24, 2024 9:19 PM
> > >
> > > On Fri, May 24, 2024 at 07:13:23AM +0000, Tian, Kevin wrote:
> > > > I'm curious to learn the real reason of that design. Is it because =
you
> > > > want to do certain load-balance between viommu's or due to other
> > > > reasons in the kernel smmuv3 driver which e.g. cannot support a
> > > > viommu spanning multiple pSMMU?
> > >
> > > Yeah, there is no concept of support for a SMMUv3 instance where it's
> > > command Q's can only work on a subset of devices.
> > >
> > > My expectation was that VIOMMU would be 1:1 with physical iommu
> > > instances, I think AMD needs this too??
> > >
> >
> > Yes this part is clear now regarding to VCMDQ.
> >
> > But Nicoline said:
> >
> > "
> > One step back, even without VCMDQ feature, a multi-pSMMU setup
> > will have multiple viommus (with our latest design) being added
> > to a viommu list of a single vSMMU's. Yet, vSMMU in this case
> > always traps regular SMMU CMDQ, so it can do viommu selection
> > or even broadcast (if it has to).
> > "
> >
> > I don't think there is an arch limitation mandating that?
>=20
> What I mean is for regular vSMMU. Without VCMDQ, a regular vSMMU
> on a multi-pSMMU setup will look like (e.g. three devices behind
> different SMMUs):
> |<------ VMM ------->|<------ kernel ------>|
>        |-- viommu0 --|-- pSMMU0 --|
> vSMMU--|-- viommu1 --|-- pSMMU1 --|--s2_hwpt
>        |-- viommu2 --|-- pSMMU2 --|
>=20
> And device would attach to:
> |<---- guest ---->|<--- VMM --->|<- kernel ->|
>        |-- dev0 --|-- viommu0 --|-- pSMMU0 --|
> vSMMU--|-- dev1 --|-- viommu1 --|-- pSMMU1 --|
>        |-- dev2 --|-- viommu2 --|-- pSMMU2 --|
>=20
> When trapping a device cache invalidation: it is straightforward
> by deciphering the virtual device ID to pick the viommu that the
> device is attached to.

I understand how above works.

My question is why that option is chosen instead of going with 1:1
mapping between vSMMU and viommu i.e. letting the kernel to
figure out which pSMMU should be sent an invalidation cmd to, as
how VT-d is virtualized.

I want to know whether doing so is simply to be compatible with
what VCMDQ requires, or due to another untold reason.

>=20
> When doing iotlb invalidation, a command may or may not contain
> an ASID (a domain ID, and nested domain in this case):
> a) if a command doesn't have an ASID, VMM needs to broadcast the
>    command to all viommus (i.e. pSMMUs)
> b) if a command has an ASID, VMM needs to initially maintain an
>    S1 HWPT list by linking an ASID when adding an HWPT entry to
>    the list, by deciphering vSTE and its linked CD. Then it needs
>    to go through the S1 list with the ASID in the command, and to
>    find all corresponding HWPTs to issue/broadcast the command.
>=20
> Thanks
> Nicolin