Date:   Fri, 10 Mar 2023 10:00:58 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     Nicolin Chen <nicolinc@nvidia.com>, robin.murphy@arm.com,
        will@kernel.org, eric.auger@redhat.com, kevin.tian@intel.com,
        baolu.lu@linux.intel.com, joro@8bytes.org,
        shameerali.kolothum.thodi@huawei.com,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org, yi.l.liu@intel.com
Subject: Re: [PATCH v1 02/14] iommufd: Add nesting related data structures
 for ARM SMMUv3
Message-ID: <ZAs4Gtuom9V8f1iW@nvidia.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <364cfbe5b228ab178093db2de13fa3accf7a6120.1678348754.git.nicolinc@nvidia.com>
 <20230309134217.GA1673607@myrica>
 <ZAnx0lUkw02cVTi+@nvidia.com>
 <ZAq3LJgbo8ApejvU@Asurada-Nvidia>
 <20230310125453.GB1745536@myrica>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230310125453.GB1745536@myrica>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1U8niqAseZYrg9/z9pDSgVRq7uy++Y5aSoZEO3wxckMIvy6hkGxjeFZbgH50?=
 =?us-ascii?Q?jfZj0+j6AcsG+d5+fWzZrWCqoP43f5C5Ut9YQXWIUHJDEKqr0HDaXymQbYW8?=
 =?us-ascii?Q?Av6SLFUOR/hOO+BwaSF1BlUQm5TchFeYly8Y61UJvmVgejJTobP8qI7Wlj/K?=
 =?us-ascii?Q?pcv0nrDF7vykl872Rfl8u05Aqcj/zmapJbPFc7kWSYE0h2etPyoH5fk+Mk07?=
 =?us-ascii?Q?1RwM0zy1H5ZNHbPdQntNDmG8FNJMpzaKNgkGuz/fQREkmK4KmWv/NyC32mC8?=
 =?us-ascii?Q?4WOO2iYrqutICasONIpIqWonpCBRlZY7RW0CG69/vK8E/opcyxng76wCjWyf?=
 =?us-ascii?Q?D10TgWLO208FFdWClGw/njSqY8HfyBleKViuElRVM0YSlB3JD03JEVwUtjDS?=
 =?us-ascii?Q?4F51FzXln5TswvbMLCpodV6iFRPI0rA7iEm3L+9ocTmJshh6c7eg1iCO7MMo?=
 =?us-ascii?Q?8Snyxs0TVC/c34LL2eymDk6+EdN7widD69B+srMUm5w8lcSBR25M46bqT0O4?=
 =?us-ascii?Q?2Ec6Por8EroBPJcQPZajgwe1suiXTYw3WkizDr34BymAMEY9BgMI7iWHHeyd?=
 =?us-ascii?Q?mc58/R9TladP1XEtH2JVTlyq2gqQcTNZTnwfwk5YL9WYTkYRq6FYd1bQrs0+?=
 =?us-ascii?Q?LdWNC79NgU5nbmHGMKyzZrfRTIaXr0JTWLKth0vATm0J07j9pp2TKa4peEs6?=
 =?us-ascii?Q?iWsT8dqVtm/Pja6y58krbhSqA38vR0PFnGy+jqJ0L6f2VVvYt0vniqP56qw/?=
 =?us-ascii?Q?PhI8+U2EqQdv869hpDZQ8qzigC1ghuMnEJuEdNXMllNBG5ZVDW7O0/nafba8?=
 =?us-ascii?Q?4N+BNy8mzsA5dca9L+mcZr4VSt04peZNjzGWVwmMo1HIIx6ERLshPK8CUHkW?=
 =?us-ascii?Q?tZa83RmL+o4cnpZl8BzQI/XmWY8GQ0WvHJjIzdx2TH2T3fNG91003lA6iU28?=
 =?us-ascii?Q?jf45hu1yBn9LrDKnG2emn/9MaLAUV98GrB8N9x2rTOhCZzcDY6Fet2oEulUk?=
 =?us-ascii?Q?tl2s6FzURNCsHT3349O0hjfp7kiPFhgryce+fyMW98uzNrvcs86tPOTH4s73?=
 =?us-ascii?Q?bSY4H30gUfWuO3gE8MkbxojP7Bmy0Z1VqdUgialtzvPfqAWSjyQIbmrKu9K2?=
 =?us-ascii?Q?mpbI8dlZO8NfgX6uxc8O2wEI0Pl9bDSyvJ2xFzK1DgTCX/QXzClJO9SuxQje?=
 =?us-ascii?Q?QUnCRftIKD8hxUY1dCKIhsZxZ3VCbf+PUuO2WOmoFiQiNkXpUFgVIiX30j/r?=
 =?us-ascii?Q?arOWqhP+1JzuNYjjn5quGTWKTvgMRwp6U7WGXMZnZ5+9CcyCY6Bxpr3+DpGL?=
 =?us-ascii?Q?JdMCUyu7kHSG27P9qdUJ45ao41xpJ9MXVpAweFfjoxcalOO7uYNHB0128etO?=
 =?us-ascii?Q?BXxxCcOrofUEx49Bai3xJ3H02BRKROWY0nfcPJvrj+s55pO12hJSJckOnZHb?=
 =?us-ascii?Q?27Zg77Rr43/615Se1aSTzlN0Zi5yIkIWGj+vw/Ha0hWFONin8xhorfORxDP0?=
 =?us-ascii?Q?evYwvKfqXC+BdEUE3ijxe+80Ed6ddBv9mOs3bqBaNvURG6cj41wkErGZ3RMC?=
 =?us-ascii?Q?n6mvK9UDPYcRnyCunG7Ckg1O1jE1+DKrqzuONfGq?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f24307a1-6cd8-4d7f-800d-08db216fe14c
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2023 14:01:01.5389
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sNDvgvAsYTJt62FENeKq+rGHUB1PuRs3yRbl+oG9dkMZBpAXCVd/b3Wq3xlSKNvm
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6747
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Mar 10, 2023 at 12:54:53PM +0000, Jean-Philippe Brucker wrote:
> On Thu, Mar 09, 2023 at 08:50:52PM -0800, Nicolin Chen wrote:
> > On Thu, Mar 09, 2023 at 10:48:50AM -0400, Jason Gunthorpe wrote:
> > 
> > > Nicolin, I think we should tweak the uAPI here so that the
> > > invalidation opaque data has a format tagged on its own, instead of
> > > re-using the HWPT tag. Ie you can have a ARM SMMUv3 invalidate type
> > > tag and also a virtio-viommu invalidate type tag.
> > 
> > The invalidation tage is shared with the hwpt allocation. Does
> > it mean that virtio-iommu won't have it's own allocation tag?
> 
> I'm not entirely sure what you mean by allocation tag. 

He means the tag identifying the allocation driver specific data is
the same tag that is passed in to identify the invalidation driver
specific data.

With the notion that the allocation data and invalidation data would
be in the same driver's format.

> Note that none of this is set in stone. It copies the Linux API we
> originally discussed, but we were waiting for progress on that front
> before committing to anything. Now we'll probably align to the new API
> where possible, leaving out what doesn't work for virtio-iommu.

IMHO virtio-iommu should stand alone and make sense with its own
internal object model.

eg I would probably try not to have guests invalidate PASID. Have a
strong ASID model and in most cases have the hypervisor track where
the ASID's are mapped to PASID/etc and rely on the hypervisor to spew
the invalidations to PASID as required. It is more abstracted from the
actual HW for the guest. The guest can simply say it changed an IOPTE
under a certain ASID.

The ugly wrinkle is SMMUv3 but perhaps your idea of allowing the
hypervisor to manage the CD table in guest memory is reasonable.

IMHO it is a missing SMMUv3 HW feature that the CD table doesn't have
the option to be in hypervisor memory. AMD allows both options - so
I'm not sure I would invest a huge amount to make special cases to
support this... Assume a SMMUv3 update might gain the option someday.

Jason