Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Thread-Topic: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Thread-Index: AQHXrSGPLoYXtOF3o0iA7Cse+/LM66uuxm8AgACjoWCAALKBgIABNQdAgAA7IoCAAAByIIAABmEAgAAAUDA=
Date:   Thu, 23 Sep 2021 12:45:17 +0000
Message-ID: <BN9PR11MB5433F297E3FA20DDC05020E18CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-12-yi.l.liu@intel.com>
 <20210921174438.GW327412@nvidia.com>
 <BN9PR11MB543362CEBDAD02DA9F06D8ED8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210922140911.GT327412@nvidia.com>
 <BN9PR11MB5433A47FFA0A8C51643AA33C8CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923120653.GK964074@nvidia.com>
 <BN9PR11MB543309C4D55D628278B95E008CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923123118.GN964074@nvidia.com>
In-Reply-To: <20210923123118.GN964074@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?bKEMgV0mi50ajXgiCxfRkcsfHZNb5Bmnq46zXcp00PKd9k2JeI7xMYn57sWj?=
 =?us-ascii?Q?GYCBQ3UHwISc86efuWblZF+vj0hOUMV1B3q/UVuu5te0RErlPBaRDyeO1U78?=
 =?us-ascii?Q?MzOcndr3V2fLnC5th7l5eP8d+zxfTO1Iy+HTQyPtgcUfLQSliyx26hY1rTi/?=
 =?us-ascii?Q?JA2+PwCx5JfCRFt6O+km3iYk3h1q4sIZm7b6TyJn11CYWd+D8UhIqB78798D?=
 =?us-ascii?Q?+eGbiHj/Kpoo8MekEluUVHcdTFGdMiiCqnsq2GLIa7UuRGLSn8pQ/BOIQOjS?=
 =?us-ascii?Q?1S4Y8pLEeYtOcMB59qOSzldATo0Jba00kpzMpr4c6C9wbmnI3Pel9r0aoi0J?=
 =?us-ascii?Q?l8+9Ha/tlkX+hoHVudLNBQc6K3F4teJY4LB0cqDr7s0znKcAjQnX82D2LZjY?=
 =?us-ascii?Q?jU4K6BO89+yaRp1dvQJPLLztL3/C5D00BFzkmOJQipp47nvCdmI2mRhB5eHy?=
 =?us-ascii?Q?kvK3d43HJ4+HyqWxU8ljfM/ICv1FBDidxD67Sgb7LBz7Q5b2QFSbq22KSZcP?=
 =?us-ascii?Q?EzuR3lj6W6BWj6Sf/cFFx9vYKDEuTsFIZ63Qn4+DrdRGVWVtKMWTxZNEr+yW?=
 =?us-ascii?Q?6Koa83eCcmy4/Vbdk2NnywS71VEea4kJbYI8pnp5Hvtkwb/B5NKlKBDdImKZ?=
 =?us-ascii?Q?AVDdSvBZ+ph/g5YwA4rntF+qzefh+/wrsLVIEIlo+uM0ekrSNk8Wq4yNwbEp?=
 =?us-ascii?Q?+eJanJzIOv/aGURiZBjyXpuu0RcEu5OLjzSntc/5GjnOCoeBT/0UBcqcvMKr?=
 =?us-ascii?Q?qTjtdD7SvIwXuZHL2b/JkxIJDzCEj/x3uty3Lm8CKqPi2gqQ/LDd9VoaFzq+?=
 =?us-ascii?Q?Tou9ir/YqhA/8LvApzC//+hS4zye2kGMb9HDVwBI0lSg3MNWV9AE2yHph81S?=
 =?us-ascii?Q?OjQDIDI5wV6gkm0kqu7qaqI85jvJfVrK3OHRGCWY1q7QcKAzN5c/qbxoWs0l?=
 =?us-ascii?Q?iUyUNlPC2Lr+AHNktdmDk4j/U0df5p8w6Ds7yVeo45FAyoWz94HtApC5xFAV?=
 =?us-ascii?Q?7BXzQtzQUUGz8IDlKTcQLd0Rz5RlY2ECIpRDXYJtTOp/YMr1PaM4PGf28GoE?=
 =?us-ascii?Q?GP/+YGpEoOkSrkjSa/xj5ycIBsuMKMN4juP0X9JFuPNCK/be0ypGI9Ph6y3x?=
 =?us-ascii?Q?AYFy8p/gnlXBQRxrRVWUuXpU6ZSL/uYgEyaD4lZ9yV5PUUvII0W+XTpGqNzu?=
 =?us-ascii?Q?VLsukX5Z/aYGMHfrJ55/3in73i5+CBOrSG9258s1mUaOTPNdzEeb3VrbXAYW?=
 =?us-ascii?Q?vV0eVN74j8zpoDwni55MNj6NOcjZitw9E/x8XE7kxUK7amiaeYOpxcn2KpSc?=
 =?us-ascii?Q?IZAYlNc6irHH3pef7vUHIg2+?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: aa10499e-1396-4a0e-3f5e-08d97e8ffef2
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2021 12:45:17.0595
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RkQR1wh/4wSJiPbIRYsUEXqe5Db/Eh05nmF39yT1nAd4lRVBSmwKRBmEqsKVQFVlVfI9Xm8MSOWuZvEBy+QYEg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR11MB5710
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, September 23, 2021 8:31 PM
>=20
> On Thu, Sep 23, 2021 at 12:22:23PM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Thursday, September 23, 2021 8:07 PM
> > >
> > > On Thu, Sep 23, 2021 at 09:14:58AM +0000, Tian, Kevin wrote:
> > >
> > > > currently the type is aimed to differentiate three usages:
> > > >
> > > > - kernel-managed I/O page table
> > > > - user-managed I/O page table
> > > > - shared I/O page table (e.g. with mm, or ept)
> > >
> > > Creating a shared ios is something that should probably be a differen=
t
> > > command.
> >
> > why? I didn't understand the criteria here...
>=20
> I suspect the input args will be very different, no?

yes, but can't the structure be extended to incorporate it?=20

>=20
> > > > we can remove 'type', but is FORMAT_KENREL/USER/SHARED a good
> > > > indicator? their difference is not about format.
> > >
> > > Format should be
> > >
> > > FORMAT_KERNEL/FORMAT_INTEL_PTE_V1/FORMAT_INTEL_PTE_V2/etc
> >
> > INTEL_PTE_V1/V2 are formats. Why is kernel-managed called a format?
>=20
> So long as we are using structs we need to have values then the field
> isn't being used. FORMAT_KERNEL is a reasonable value to have when we
> are not creating a userspace page table.
>=20
> Alternatively a userspace page table could have a different API

I don't know. Your comments really confused me on what's the right
way to design the uAPI. If you still remember, the original v1 proposal
introduced different uAPIs for kernel/user-managed cases. Then you
recommended to consolidate everything related to ioas in one allocation
command.

Can you help articulate the criteria first?

>=20
> > yes, the user can query the permitted range using DEVICE_GET_INFO.
> > But in the end if the user wants two separate regions, I'm afraid that
> > the underlying iommu driver wants to know the exact info. iirc PPC
> > has one global system address space shared by all devices. It is possib=
le
> > that the user may want to claim range-A and range-C, with range-B
> > in-between but claimed by another user. Then simply using one hint
> > range [A-lowend, C-highend] might not work.
>=20
> I don't know, that sounds strange.. In any event hint is a hint, it
> can be ignored, the only information the kernel needs to extract is
> low/high bank?

iirc Dave said that the user needs to claim a range explicitly. 'claim'
sounds not a hint to me. Possibly it's time for Dave to chime in.=20

>=20
> > yes PPC can use different format, but I didn't understand why it is
> > related user-managed page table which further requires nesting. sound
> > disconnected topics here...
>=20
> It is just a way to feed through more information if we get stuck
> someday.

You mean that we should define uAPI for all future possible extensions
now to minimize the frequency of changing it?

>=20
> > > ARM *does* need PASID! PASID is the label of the DMA on the PCI bus,
> > > and it MUST be exposed in that format to be programmed into the PCI
> > > device itself.
> >
> > In the entire discussion in previous design RFC, I kept an impression t=
hat
> > ARM-equivalent PASID is called SSID. If we can use PASID as a general
> > term in iommufd context, definitely it's much better!
>=20
> SSID is inside the chip and part of the IOMMU. PASID is part of the
> PCI spec.
>=20
> iommufd should keep these things distinct.
>=20
> If we are talking about a PCI TLP then the name to use is PASID.

If Jean doesn't object...

>=20
> > > All of this should be able to support a userspace, like DPDK, creatin=
g
> > > a PASID on its own without any special VFIO drivers.
> > >
> > > - Open iommufd
> > > - Attach the vfio device FD
> > > - Request a PASID device id
> > > - Create an ios against the pasid device id
> > > - Query the ios for the PCI PASID #
> > > - Program the HW to issue TLPs with the PASID
> >
> > this all makes me very confused, and completely different from what
> > we agreed in previous v2 design proposal:
> >
> > - open iommufd
> > - create an ioas
> > - attach vfio device to ioasid, with vPASID info
> > 	* vfio converts vPASID to pPASID and then call
> iommufd_device_attach_ioasid()
> > 	* the latter then installs ioas to the IOMMU with RID/PASID
>=20
> This was your flow for mdev's, I've always been talking about wanting
> to see this supported for all use cases, including physical PCI
> devices w/ PASID support.

this is not a flow for mdev. It's also required for pdev on Intel platform,
because the pasid table is in HPA space thus must be managed by host=20
kernel. Even no translation we still need the user to provide the pasid inf=
o.

>=20
> A normal vfio_pci userspace should be able to create PASIDs unrelated
> to the mdev stuff.
>=20
> > > AFAICT I think it is the former in the Intel scheme as the "vPASID" i=
s
> > > really about presenting a consistent IOMMU handle to the guest across
> > > migration, it is not the value that shows up on the PCI bus.
> >
> > It's the former. But vfio driver needs to maintain vPASID->pPASID
> > translation in the mediation path, since what guest programs is vPASID.
>=20
> The pPASID definately is a PASID as it goes out on the PCIe wire
>=20
> Suggest you come up with a more general name for vPASID?
>=20

as explained earlier, on Intel platform the user always needs to provide=20
a PASID in the attaching call. whether it's directly used (for pdev)
or translated (for mdev) is the underlying driver thing. From kernel
p.o.v, since this PASID is provided by the user, it's fine to call it vPASI=
D
in the uAPI.

Thanks
Kevin