From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "peter.maydell@linaro.org" <peter.maydell@linaro.org>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Jean-Philippe Brucker <jean-philippe@linaro.org>,
        "maz@kernel.org" <maz@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "vsethi@nvidia.com" <vsethi@nvidia.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "wangxingang5@huawei.com" <wangxingang5@huawei.com>,
        "vivek.gautam@arm.com" <vivek.gautam@arm.com>,
        "zhangfei.gao@linaro.org" <zhangfei.gao@linaro.org>,
        "eric.auger.pro@gmail.com" <eric.auger.pro@gmail.com>,
        "will@kernel.org" <will@kernel.org>,
        "kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>
Subject: RE: [RFC v16 1/9] iommu: Introduce attach/detach_pasid_table API
Thread-Topic: [RFC v16 1/9] iommu: Introduce attach/detach_pasid_table API
Thread-Index: AQHXyx+9zoVf6ec36EaoAw6mFm2RjKwlh1WAgAGLFwCAARJNAIAAUNaAgABaKgCAAEnegIAAE6oAgACHsjCAABVZoIAAzVeAgAED4rA=
Date:   Fri, 10 Dec 2021 08:56:56 +0000
Message-ID: <BN9PR11MB527612D1B4E0DC85A442D87D8C719@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20211027104428.1059740-2-eric.auger@redhat.com>
 <Ya3qd6mT/DpceSm8@8bytes.org>
 <c7e26722-f78c-a93f-c425-63413aa33dde@redhat.com>
 <e6733c59-ffcb-74d4-af26-273c1ae8ce68@linux.intel.com>
 <fbeabcff-a6d4-dcc5-6687-7b32d6358fe3@redhat.com>
 <20211208125616.GN6385@nvidia.com> <YbDpZ0pf7XeZcc7z@myrica>
 <20211208183102.GD6385@nvidia.com>
 <BN9PR11MB527624080CB9302481B74C7A8C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <BN9PR11MB5276D3B4B181F73A1D62361C8C709@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20211209160803.GR6385@nvidia.com>
In-Reply-To: <20211209160803.GR6385@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LvGEBvzqIpYDFaO7vy0ogmz4glXqWIqyNXp9h+LAZCNOKQeVVAPy6Zc8xW+f?=
 =?us-ascii?Q?nfOrEB4oMq5pRNoSG1Y6+9GsXHIenYwmeRp2xNM2OvEacvGP0Zjz7yNuODvp?=
 =?us-ascii?Q?R9+XK0qHNA86/fX8KmK2qsqAbwfRNC1ABHnSEq3ktFXqpnl0ScRlJtrTj6kx?=
 =?us-ascii?Q?6spvaRdfniS3MyeN6DrTUQZSoLmUQlwrVp/oFhHRV94SqUTblSHCeOcBJ9L6?=
 =?us-ascii?Q?+XFhwzMD3wOdTYNgTAe86gudmjXUdDqyZDQ4ZeZhQZ3SliADxx6HW1Luzx4n?=
 =?us-ascii?Q?WuFrsW2PbBlbEuFyc83RU6RLezW2Oom2GD52oBsK6ND4KxBaODLTL357DvMX?=
 =?us-ascii?Q?UQSs0Gep4I85SYPJ/v2/x/74O6JrQhIdinql1ra1/n5Jw4TUm6TOQfbVTb1f?=
 =?us-ascii?Q?oVa434RDBfUMeEeQBAhxBGGqErGHyqwuN+VeDpWdUqznpn2l2K9WWS+eAkbi?=
 =?us-ascii?Q?tBPGynFFRdR4QoR9ttaA+IS5HNhr4n8lAZIaHMBavRoU0y6R9byjmkhkCMZm?=
 =?us-ascii?Q?hRjxAq7YIJIxqeWVkmfh29aW1fU/g4vrmw7AvAOngBwH1+g66eNuEuM7Z71C?=
 =?us-ascii?Q?3AIQC6eCWFImGiJ57WOOS05W8LWE+5Ib6liap1sq5vuiK/GlRciA+MY/iCiG?=
 =?us-ascii?Q?FvCjIpLifCUQafD/V/peNedEQIrxSeFia2Driftn3A9voroo03TnPg8p8KPz?=
 =?us-ascii?Q?xwBDXO+pufwhK03hw65zgrM/TEoADoVsUQSQLP6cOs19rEvOA/Q8hSQKQ+Ks?=
 =?us-ascii?Q?KJAOnRwsBf1k89AI9tdHM/tqY9VRkxCaubcoHHpQ90UKoh/N0bIy6NqD65ZJ?=
 =?us-ascii?Q?9reNZ2Cyo+bHxJgkmavXxqrHMXKJXf9M31ZAIw99XkCUTIBYQE7E50XzJhVM?=
 =?us-ascii?Q?grinXXD2ZM3KWsXUqmpN74nR0/tqThxoV9W4rQUJ63X9CcMT14k2Ue/SLRU8?=
 =?us-ascii?Q?QF3qhnuc06iGULZay2fRCpLWZt1NCR3E/9ALctNz/h8NU2yJdrzNUxWtPXcw?=
 =?us-ascii?Q?b3mvjCgOReun2xQePVPfdBaqiyk9QuaP9mh6ZHDDYIj2pGxUcTXXGOz96CyW?=
 =?us-ascii?Q?Zk250lGsJBNggTQp53vEz/63LJIvCUQMp3XQrXXhfzN5aCFndhE2vdC/lXYQ?=
 =?us-ascii?Q?NntybhDHI1TbOJi/WBxDD5+e0tkvcf14NNwv7hwBnvvK6TVaVxgDrPOzQIdf?=
 =?us-ascii?Q?dt+7ezTGvI4Tr5iuTGQHSU6sTJON4RX8AWRT+ipaYQQkPFya1detZjUejFN1?=
 =?us-ascii?Q?t77i/E0W2OCxXvJ4C2rsxg29Ul14Tj7zXGITAVuatWy8ApAHgn01yXHglqaa?=
 =?us-ascii?Q?zfLD92k5iQmk8El4Uvf0XDoki6jaUk3bxVS0stZLZXiXOhVmJ7Gaa8kx88ly?=
 =?us-ascii?Q?/dai/i1nj0bu72VxQZ32SK1nOBdhnDH8Y9n4RvIz0gMY7Z6kyszMkRO1djX+?=
 =?us-ascii?Q?otXt2POoRFLeFTqE0DnMwe4tJ6mdwWqjBN5RBQ15YNPE2hzKlNZvazU5099E?=
 =?us-ascii?Q?wlyUxDkET3zbZmTEysKygSisC7KjjWMgB6BMaqT2MOloo+qnKy3MuytD+yTC?=
 =?us-ascii?Q?bEJXZXqOFdXWrfL9pikGMLm8r9z0aIXqXqi/M45d2I9w0jDsG2bah61eGPoG?=
 =?us-ascii?Q?20HHR02GTQDm1Ms5//JBu6M=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 44878884-d19a-4ceb-ccab-08d9bbbb0500
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Dec 2021 08:56:56.5020
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 9WZ1e5RIZQ7tbXVb+t7hFGHhsgbOx1eDNK0CmOuv3K9TRu8fWT5CtKXrqG7Sbu/YILChYekrHKMzQ6cvASLYbQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1652
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe via iommu
> Sent: Friday, December 10, 2021 12:08 AM
>=20
> On Thu, Dec 09, 2021 at 03:59:57AM +0000, Tian, Kevin wrote:
> > > From: Tian, Kevin
> > > Sent: Thursday, December 9, 2021 10:58 AM
> > >
> > > For ARM it's SMMU's PASID table format. There is no step-2 since PASI=
D
> > > is already within the address space covered by the user PASID table.
> > >
> >
> > One correction here. 'no step-2' is definitely wrong here as it means
> > more than user page table in your plan (e.g. dpdk).
> >
> > To simplify it what I meant is:
> >
> > iommufd reports how many 'user page tables' are supported given a devic=
e.
> >
> > ARM always reports only one can be supported, and it must be created in
> > PASID table format. tagged by RID.
> >
> > Intel reports one in step1 (tagged by RID), and N in step2 (tagged by
> > RID+PASID). A special flag in attach call allows the user to specify th=
e
> > additional PASID routing info for a 'user page table'.
>=20
> I don't think 'number of user page tables' makes sense
>=20
> It really is 'attach to the whole device' vs 'attach to the RID' as a
> semantic that should exist
>=20
> If we imagine a userspace using kernel page tables it certainly makes
> sense to assign page table A to the RID and page table B to a PASID
> even in simple cases like vfio-pci.
>=20
> The only case where userspace would want to capture the entire RID and
> all PASIDs is something like this ARM situation - but userspace just
> created a device specific object and already knows exactly what kind
> of behavior it has.
>=20
> So, something like vfio pci would implement three uAPI operations:
>  - Attach page table to RID
>  - Attach page table to PASID
>  - Attach page table to RID and all PASIDs
>    And here 'page table' is everything below the STE in SMMUv3
>=20
> While mdev can only support:
>  - Access emulated page table
>  - Attach page table to PASID

mdev is a pci device from user p.o.v, having its vRID and vPASID. From
this angle the uAPI is no different from vfio-pci (except the ARM one):

  - (sw mdev) Attach emulated page table to vRID (no iommu domain)
  - (hw mdev) Attach page table to vRID (mapped to mdev PASID)
  - (hw mdev) Attach page table to vPASID (mapped to a fungible PASID)

>=20
> It is what I've said a couple of times, the API the driver calls
> toward iommufd to attach a page table must be unambiguous as to the
> intention, which also means userspace must be unambiguous too.
>=20

No question on the unambiguous part. But we also need to consider
the common semantics that can be abstracted.

From user p.o.v a vRID can be attached to at most two page tables (if
nesting is enabled). This just requires the basic attaching form for=20
either one page table or two page tables:

	at_data =3D {
		.iommufd	=3D xxx;
		.pgtable_id	=3D yyy;
	};
	ioctl(device_fd, VFIO_DEVICE_ATTACH_PGTABLE, &at_data);

This can already cover ARM's requirement. The user page table
attached to vRID is in vendor specific format, e.g. either ARM pasid=20
table format or Intel stage-1 format. For ARM pasid_table + underlying=20
stage-1 page tables can be considered as a single big paging structure.

From this angle I'm not sure the benefit of making a separate uAPI=20
just because it's a pasid table for ARM.

Then when PASID needs to be explicitly specified (e.g. in Intel case):

	at_data =3D {
		.iommufd	=3D xxx;
		.pgtable_id	=3D yyy;
		.flags 		=3D VFIO_ATTACH_FLAGS_PASID;
		.pasid		=3D zzz;
	};
	ioctl(device_fd, VFIO_DEVICE_ATTACH_PGTABLE, &at_data);

Again, I don't think what a simple flag can solve needs to be made
into a separate uAPI.

Is modeling like above considered ambiguous?

Thanks
Kevin