Received-SPF: pass (google.com: domain of linux-kernel+bounces-178871-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Tue, 14 May 2024 12:38:57 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com,
	suravee.suthikulpanit@amd.com, joro@8bytes.org,
	linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
	linux-arm-kernel@lists.infradead.org, linux-tegra@vger.kernel.org,
	yi.l.liu@intel.com, eric.auger@redhat.com, vasant.hegde@amd.com,
	jon.grimm@amd.com, santosh.shukla@amd.com, Dhaval.Giani@amd.com,
	shameerali.kolothum.thodi@huawei.com
Subject: Re: [PATCH RFCv1 05/14] iommufd: Add IOMMUFD_OBJ_VIOMMU and
 IOMMUFD_CMD_VIOMMU_ALLOC
Message-ID: <ZkOFkfHhG2h2fv/c@nvidia.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <3aa9bc1df6a2ee58a03c6ea6ededbc210a2d23a8.1712978212.git.nicolinc@nvidia.com>
 <ZkDR4Rp57cy9qSqP@nvidia.com>
 <ZkGYN36N7sh2vUmv@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZkGYN36N7sh2vUmv@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?lxsi2Ly7WKXuEQLOJNQtVfXpAU+Bpdz4EZgGN4Y55MyHJDyl86c9nUZK50zQ?=
 =?us-ascii?Q?kxbhbJMnOgFTsux2O4d7eMZNbzI8GZQXEHhW0+McvRFlZ2Hmgdrr3miMNYFl?=
 =?us-ascii?Q?gLV55CG8Dv/xk3abG140rDSi+yMiSFendau/lwHZ4TNUhIK01YruR4PPhus5?=
 =?us-ascii?Q?YfnqN3x/Rd2FDaarC54AfsKQTYSv91CdRAh01XzNk+GxhLgQ4s7Ip4t/HebH?=
 =?us-ascii?Q?omgF0OezOGQFbm8simZx45d6XqPb0MHxQGRHApBvJmzv5neBqZqop6q1X+LU?=
 =?us-ascii?Q?yCED152sqGzNq8mnzgQmU9GxdR+TYZ0RehSm1MwBx9d1S5MREPRelStaMQKi?=
 =?us-ascii?Q?PiZ3glChDJbgn/Wk6kYglim35589PuE05OGXUUxM20XER/4q1sbBd9elc90j?=
 =?us-ascii?Q?bmrN//tl7b5zSW6C9Gt5O5ZQxMCxb/qRIaX3Q/NB4fwruGv1gmPnY9lEoTRQ?=
 =?us-ascii?Q?xndBo8jxFmrqkOD7BBu0akEIEdTHIV4D5ZD8LHw1aEKUeSCaaK2m39P0dRS5?=
 =?us-ascii?Q?0AqZfRNFTbaEOMHBH7NJr+Iqe/YZTPDb2afm6o7N04/4p13mPYNaGjeU6BrC?=
 =?us-ascii?Q?xly24OrCTqJE4EdNl2mmm/Dd6px85Pbva0hKWisSCKSuN7wDx9/gWk73fZxt?=
 =?us-ascii?Q?Kknd7InFGrc2HhenuJIWCyflrjM2Qt90uX2KSx1F9OM4qCtvJwwjwAxcmFHl?=
 =?us-ascii?Q?cr3kiwF4Np4+jNM8DQ2QkgZtiCRgMEvVLfY6reZeDQ2fUmRlWel909isUT4v?=
 =?us-ascii?Q?Oz7f0bsHHyDciHrQ8ccsOnl48aIo7996UQv3pYLA0PkxQXbm4I7mTYlF1+vO?=
 =?us-ascii?Q?1KnaJwYiFoaAmeEqOUNCY+ctDESZUsrF7Adr89rWZF/xDD9JBICFam5kpGb+?=
 =?us-ascii?Q?j15AakqAygLYy+VGHYAy+UEtR43hG9nHKpUTMNfVlVq/OfWwo9OA7HCr0BNQ?=
 =?us-ascii?Q?QNY3+u5IWBwCj8HWGCxnXosDkILV/QaspfKDGaj2LAy8tjHlXUUGqFa8u9sK?=
 =?us-ascii?Q?zQ0v+gPJ4YZS590F2WMbGpgTGxaUm6MY1+OuI9leeZldxWXbV6k1wfUjTUDF?=
 =?us-ascii?Q?WIzPvRRhunxGondxL/Oj7m7w56xleU8wNGRalvlfvqvsrcDE8e2gZi25csJt?=
 =?us-ascii?Q?bbpabY1B33f1zF8DUiUFeKntpPUfdmdTETAKwlV9jP7DMexl0vS7PL35e3Xr?=
 =?us-ascii?Q?3X445dWiWUT9q22MnbTyMgTWzN5DbwSLsOvprx/Fe6Gn27iA/deuoxXtA5cH?=
 =?us-ascii?Q?TXMtlPfrN3jJ5nUOKPdLUOW1OrEkKiB3LQrpGsQtPVd9p7v3j5EgRN5er25I?=
 =?us-ascii?Q?zAJFAYFMbxwdDc8tHncwJPWnLnmcAGZnF5+hkI40kSqgNTx4eAXB6602soUF?=
 =?us-ascii?Q?IVFD4sPPqXOOCoYupyQzp0Ggvr6Usnxdr8SU7ZWTAQbttFH4972KAWmL4N/c?=
 =?us-ascii?Q?Uxp7BYng2gvlyJtf6RDv8fIi9aKVZRRrmy13u2GiodnZ+A3z1NVmd23wYNB7?=
 =?us-ascii?Q?EgGjtjHCqyuImrvh4yJdJNWt9L/wxGCpcR2vtkifo7Bx9p1WsK/LxeEdRY0p?=
 =?us-ascii?Q?bibNG6rLm53kT+/QSBE=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 28e61668-e0e6-4590-2d3b-08dc742bf95e
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2024 15:38:59.7264
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lrZmOq78HWMJGhxsvRFM2CnOeoNun2QPYHAxgN35oHI4cgUKnZLNxzAXCA9Ek8jg
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6146

> > > +
> > > +/**
> > > + * enum iommu_viommu_type - VIOMMU Type
> > > + * @IOMMU_VIOMMU_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension for SMMUv3
> > > + */
> > > +enum iommu_viommu_type {
> > > +	IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
> > > +};
> > 
> > At least the 241 line should be in a following patch
> 
> It's for the "enum iommu_viommu_type" mentioned in the following
> structure. Yi told me that you don't like an empty enum, and he
> did something like this in HWPT_INVALIDATE series:
> https://lore.kernel.org/linux-iommu/20240111041015.47920-3-yi.l.liu@intel.com/

I suspect 0 should be reserved as a non-set value for some
basic sanity in all these driver type enums.

Jason

> > So, to make this all work perfectly we need approx the following
> >  - S2 sharing across instances in ARM - meaning the VMID is allocated
> >    at attach not domain alloc
> >  - S2 hwpt is refcounted by the VIOMMU in the iommufd layer
> >  - VIOMMU is refcounted by every nesting child in the iommufd layer
> >  - The nesting child holds a pointer to both the S2 and the VIOMMU
> >    (viommu optional)
> >  - When the nesting child attaches to a device the STE will source the
> >    VMID from the VIOMMU if present otherwise from the S2
> >  - "RID" attach (ie naked S2) will have to be done with a Nesting
> >    Child using a vSTE that indicates Identity. Then the attach logic
> >    will have enough information to get the VMID from the VIOMMU
> 
> What is this RID attach (naked S2) case? S1DSS_BYPASS + SVA?

No, when the guest installs a vSTE that simply says bypass with no CD
table pointer. That should result in a pSTE that is the S2 with on CD
pointer.

I was originally thinking that the VMM would simply directly attach
the S2 HWPT in this caes, but given the above issue with the VMID lifetime
it makes more sense to 'attach' the viommu which holds the correct
VMID. 

The issue with direct attach the S2 HWPT is the VMID lifetime, as it
would have to borrow the VMID from the viommu but then the lifetime
becomes more complex as it has to live beyond VIOMMU destruction. Not
unsolvable but it seems easier to just avoid it entirely.

> >  - In full VIOMMU mode the S2 will never get a VMID of its own, it
> >    will always use the VIOMMU. Life cycle is simple, the VMID is freed
> >    when the VIOMMU is freed. That can't happen until all Nesting
> >    Children are freed. That can't happen until all Nesting Children
> >    are detached from devices. Detatching removes the HW touch of the VMID.
> 
> So, each VM will have one S2 HWPT/domain/iopt, but each VM can
> have multiple VIOMMU instances sharing that single S2 HWPT, and
> each VIOMMU instance (in the SMMU driver at least) holds a vmid.

Yes, right. We really want to share the S2 across instances in the end
and I have made the VMID per-instance along with the per-instance
ASID. So the above sounds like it could work

Jason