Received-SPF: pass (google.com: domain of linux-kernel+bounces-198034-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Sat, 1 Jun 2024 18:45:01 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"linux-tegra@vger.kernel.org" <linux-tegra@vger.kernel.org>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
	"jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
	"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Message-ID: <ZluWXYQjroz0fqqn@nvidia.com>
References: <BN9PR11MB5276A897A5386DFAC9A35F9D8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAoN3s96HL7ROth@nvidia.com>
 <BN9PR11MB52762A0BC67B64D4171480C78CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240524131912.GT20229@nvidia.com>
 <BN9PR11MB5276BBD592021507C3A0EBB38CF02@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlY886FT3HHLC8Cf@nvidia.com>
 <BN9PR11MB52762EA9B444DA71F551C3868CF22@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlafEktUu5znDsNt@Asurada-Nvidia>
 <BN9PR11MB5276AC43120376A2502D3D148CF32@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlfPPwsnY6P4SoGF@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZlfPPwsnY6P4SoGF@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?BDHGH+/R5hRIslQtzdnYBfXMnJJuU4UfXtJiFH/WY0x9r71sf0X/EeffuQIx?=
 =?us-ascii?Q?2ZRl/9SeEwqzddQuWH8caStlM/u/fx3cPGSGAQRzGcERXNaMrSDyhgvDuVEN?=
 =?us-ascii?Q?I/g/sXPH1RE3mhhY3CUudJ9Wqqa8tpvT0ccSlLfluFripdlRJbdWPoOEssB0?=
 =?us-ascii?Q?xvg4DPWtY5cwUIov+pntsYop0CF9opzcJwY+pIp2jO1t+D/bZHRQwromqbfV?=
 =?us-ascii?Q?0NsLIdh7clYrSScnYzkMLtonpDPN9ZoBmevMvakdWN9jzx3xsATLgfCNsj2Q?=
 =?us-ascii?Q?1HJGXb9ryAJYwdpT+dvGyPG4QueNlgiDI9UcFivg1X8JvesFT3F+Djawycox?=
 =?us-ascii?Q?Gc+swTh514P3CUDDdup8g2uDhnANSoJ1lnPPQ/9nEKa3Bb8WYZbERhhO3eJX?=
 =?us-ascii?Q?fX5HeSgFSrLemr6YxPqdVU561l+qJNdzLiQ4OXOPr/7VXnEBC/U8fPz7IbG/?=
 =?us-ascii?Q?DQIct26r0W2pGjLwXtRW8uVgeVQ60it/Apq6Ij4Iw1ZfcvjmV/ytUHwfGxmK?=
 =?us-ascii?Q?zhnm40HXbpxohRCIgs2+MnNXvRtu1K4w2o3BzvdDxQfU3dEsfJN2vm1ZVCJe?=
 =?us-ascii?Q?bFVujfosS/36ZEuhpcKfyOhAy4caRuIJ+Kmj2+Eq07dOZHjqK9Nnv/RwtoxP?=
 =?us-ascii?Q?DiAiac+8UrXMq0Mndyq85aypTWocDtPcArn32OCYgqDFCIcPeAUEETJ2rEV9?=
 =?us-ascii?Q?WKzYP2FWA0+atmoQu+4nNgyDckbDdFA8aZ9Kut2fCizeI324FPnkPTzv8G0V?=
 =?us-ascii?Q?WiC12F2u5hQL3b3Bd98XklPNQjhIMHPqoBpkAggP0+j0XyB3Iqbx1tFv92WA?=
 =?us-ascii?Q?11j4niFlXjfqKUr4f4omyU6TOWbZ8QgKPifXawwdtF7ceK8inHDYU1Ssdwv4?=
 =?us-ascii?Q?wA+vWnUNwO2PmxqayNW/DSnO6Os/ZZXYPNZk2SMAz/AZNCAOCthO0QwEk84B?=
 =?us-ascii?Q?0YhHNvFZTGfe7FrYJ5cQwZWANJu3PZXAqVgST51dq5tPvBootWiWrjPQt826?=
 =?us-ascii?Q?snxVMfVnp3uFZwhcgUXHgFlxoIStTwoSoU0znSZRLLirMg4ZTfZH2WSAGFnz?=
 =?us-ascii?Q?XhiGDECPaPPyehjFKArrj2xtZEhwv41cOXKu/dpH5bKXffl89kBYipzShKyr?=
 =?us-ascii?Q?ukVg9QP5jKY/0IikhmodWZ2CXpN/RbYvrnlaxZH9YXuZfIab60BTzpDB1pIT?=
 =?us-ascii?Q?Yl7CHi0wZIr+FjqewU68atsHpuG+E59P5IKE0PAd0Rh5mloubvuNW6ct9I0V?=
 =?us-ascii?Q?38v3F9DTaIOeH1xBlT0s9D+DK8NLfyjAkMiOdBRLX0d1RyRH6Zh5kxGcrSc2?=
 =?us-ascii?Q?R0TcEvr2f4sRE923dOggVdfUYeMsopKDfzsGKkqMk8YGdFvUraxtAq7mYEFt?=
 =?us-ascii?Q?1VirZLCznI4N9d+uUZ4NxTGvcSWvGdkLMmy8eoqdEkZknjuEO/v84LyY6A64?=
 =?us-ascii?Q?jPv3D1sBqs+xCYNUlB5HVpI4f1q4oLN5IH46vrbj9ojhYmsyp0APfiXncJjz?=
 =?us-ascii?Q?9rw3lYv81IH6xvK5Iv6BhqZ3E0Ls5qngDK+jFsHP6+pXW0wz2duAk64gBjhL?=
 =?us-ascii?Q?+RFKcDDaimGUUqwR+to=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 597b857d-8a66-407b-ca41-08dc828f52b3
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2024 23:05:26.0284
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YQjkZKnJ2dsqnkPF+528MQ/wbRKTo37NQm4fq9qsBvEs5Mwblrdg8nlnDG654JuW
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8906

On Wed, May 29, 2024 at 05:58:39PM -0700, Nicolin Chen wrote:
> On Thu, May 30, 2024 at 12:28:43AM +0000, Tian, Kevin wrote:
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > Sent: Wednesday, May 29, 2024 11:21 AM
> > > On Wed, May 29, 2024 at 02:58:11AM +0000, Tian, Kevin wrote:
> > > > My question is why that option is chosen instead of going with 1:1
> > > > mapping between vSMMU and viommu i.e. letting the kernel to
> > > > figure out which pSMMU should be sent an invalidation cmd to, as
> > > > how VT-d is virtualized.
> > > >
> > > > I want to know whether doing so is simply to be compatible with
> > > > what VCMDQ requires, or due to another untold reason.
> > >
> > > Because we use viommu as a VMID holder for SMMU. So a pSMMU must
> > > have its own viommu to store its VMID for a shared s2_hwpt:
> > >         |-- viommu0 (VMIDx) --|-- pSMMU0 --|
> > >  vSMMU--|-- viommu1 (VMIDy) --|-- pSMMU1 --|--s2_hwpt
> > >         |-- viommu2 (VMIDz) --|-- pSMMU2 --|
> > >
> > 
> > there are other options, e.g. you can have one viommu holding multiple
> > VMIDs each associating to a pSMMU.
> 
> Well, possibly. But everything previously in a viommu would have
> to be a list (for number of VMIDs) in the kernel level: not only
> a VMID list, but also a 2D virtual ID lists, something like:
> 
> struct xarray vdev_ids[num_of_vmid]; // per-IOMMU vID to pID lookup

I feel it makes most sense that ARM (and maybe everyone) just have a
viommu per piommu.

The main argument against is we haven't made it efficient for the VMM
to support multiple piommus. It has to do a system call per piommu
each time it processes the cmdq.

But, on the other hand, if you care about invalidation efficiency it
is kind of silly not to expose the piommus to the guest so that the
invalidation scope can be appropriately narrowed. Replicating all ASID
invalidations to all piommus doesn't make alot of sense if the guest
can know that only one piommu actually needs invalidation.

> And a driver in this case would be difficult to get a complete
> concept of a viommu object since it's core object and shared by
> all kernel-level IOMMU instances. If a driver wants to extend a
> viommu object for some additional feature, e.g. VINTF in this
> series, it would likely have to create another per-driver object
> and again another list of this kind of objects in struct viommu.

Right, we need some kind of per-piommu object because we have
per-piommu data.

> Oh. With regular nested SMMU, there is only one virtual SMMU in
> the guest VM. No need of copying physical topology. Just the VMM
> needs to allocate three viommus to add them to a list of its own.

I understand the appeal of doing this has been to minimize qemu
changes in its ACPI parts, if we tackle that instead maybe we should
just not implement viommu to multiple piommu. It is somewhat
complicated.

Jason