Received-SPF: pass (google.com: domain of linux-kernel+bounces-193399-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.160 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C
Date: Tue, 28 May 2024 20:20:50 -0700
From: Nicolin Chen <nicolinc@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
CC: Jason Gunthorpe <jgg@nvidia.com>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Message-ID: <ZlafEktUu5znDsNt@Asurada-Nvidia>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <c97a98a72ee3498c587e5898d6b899553ccd9b27.1712978212.git.nicolinc@nvidia.com>
 <BN9PR11MB5276A897A5386DFAC9A35F9D8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAoN3s96HL7ROth@nvidia.com>
 <BN9PR11MB52762A0BC67B64D4171480C78CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240524131912.GT20229@nvidia.com>
 <BN9PR11MB5276BBD592021507C3A0EBB38CF02@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlY886FT3HHLC8Cf@nvidia.com>
 <BN9PR11MB52762EA9B444DA71F551C3868CF22@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52762EA9B444DA71F551C3868CF22@BN9PR11MB5276.namprd11.prod.outlook.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 May 2024 03:21:07.8449
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: ea5b4631-7ccb-46c5-e081-08dc7f8e61ad
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource:
	BL6PEPF0001AB53.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9121

On Wed, May 29, 2024 at 02:58:11AM +0000, Tian, Kevin wrote:
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Wednesday, May 29, 2024 4:23 AM

> > What I mean is for regular vSMMU. Without VCMDQ, a regular vSMMU
> > on a multi-pSMMU setup will look like (e.g. three devices behind
> > different SMMUs):
> > |<------ VMM ------->|<------ kernel ------>|
> >        |-- viommu0 --|-- pSMMU0 --|
> > vSMMU--|-- viommu1 --|-- pSMMU1 --|--s2_hwpt
> >        |-- viommu2 --|-- pSMMU2 --|
> >
> > And device would attach to:
> > |<---- guest ---->|<--- VMM --->|<- kernel ->|
> >        |-- dev0 --|-- viommu0 --|-- pSMMU0 --|
> > vSMMU--|-- dev1 --|-- viommu1 --|-- pSMMU1 --|
> >        |-- dev2 --|-- viommu2 --|-- pSMMU2 --|
> >
> > When trapping a device cache invalidation: it is straightforward
> > by deciphering the virtual device ID to pick the viommu that the
> > device is attached to.
> 
> I understand how above works.
> 
> My question is why that option is chosen instead of going with 1:1
> mapping between vSMMU and viommu i.e. letting the kernel to
> figure out which pSMMU should be sent an invalidation cmd to, as
> how VT-d is virtualized.
> 
> I want to know whether doing so is simply to be compatible with
> what VCMDQ requires, or due to another untold reason.

Because we use viommu as a VMID holder for SMMU. So a pSMMU must
have its own viommu to store its VMID for a shared s2_hwpt:
        |-- viommu0 (VMIDx) --|-- pSMMU0 --|
 vSMMU--|-- viommu1 (VMIDy) --|-- pSMMU1 --|--s2_hwpt
        |-- viommu2 (VMIDz) --|-- pSMMU2 --|

Thanks
Nicolin