Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Alex Williamson <alex.williamson@redhat.com>
CC:     Jason Gunthorpe <jgg@nvidia.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.org>,
        David Gibson <david@gibson.dropbear.id.au>,
        Jason Wang <jasowang@redhat.com>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "Enrico Weigelt, metux IT consult" <lkml@metux.net>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Shenming Lu <lushenming@huawei.com>,
        Joerg Roedel <joro@8bytes.org>,
        Eric Auger <eric.auger@redhat.com>,
        "Jonathan Corbet" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        "Kirti Wankhede" <kwankhede@nvidia.com>,
        Robin Murphy <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "David Woodhouse" <dwmw2@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>,
        "Lu Baolu" <baolu.lu@linux.intel.com>
Subject: RE: [RFC v2] /dev/iommu uAPI proposal
Thread-Topic: [RFC v2] /dev/iommu uAPI proposal
Thread-Index: Add0lrMH87IsTsl5Rp6WN1oQU6kGMQAdcmAAAGvGIGAAJH+YAAAJ2M1w
Date:   Mon, 12 Jul 2021 23:41:21 +0000
Message-ID: <BL1PR11MB54299AF6F55B6055A4E3ABA58C159@BL1PR11MB5429.namprd11.prod.outlook.com>
References: <BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com>
        <20210709155052.2881f561.alex.williamson@redhat.com>
        <BN9PR11MB54336FB9845649BB2D53022C8C159@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210712124150.2bf421d1.alex.williamson@redhat.com>
In-Reply-To: <20210712124150.2bf421d1.alex.williamson@redhat.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.5.1.3
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?TYVNihxT2MiTzLPI+tczxTiDiPfjk+h+Lw1qXb6hrCUzeIolUh8shnAVs9Eg?=
 =?us-ascii?Q?jsby7gQHduRbRppUAuGdMs6tWIqznL5IkGGh81YvS7DcWxfjKmEVZlyHFcPi?=
 =?us-ascii?Q?vCNMtcxQ1fqixf3+edpRZy1y74rLtUrjqFVX0RYxGAGkkCbZ7gDrtkZ3p0l2?=
 =?us-ascii?Q?1fhlF9yLZehLCIGB4R9HtVVWGFfEYaqpQj62i/wTSYqGoBSDUc3cpDokWKRA?=
 =?us-ascii?Q?B6OabssvFLHEJWNnxSRNQqphh5SaaCz+4kAB6zIBoD902NZGGs6Xk4WMjFmr?=
 =?us-ascii?Q?/z81J9+IxCM/TwIgjOysqvlq7McRcUVBVbr0eplY13Rv+KwpLCrqlcKslfpH?=
 =?us-ascii?Q?mB9A+Tk67yuI7uuqHoJq7kgixp3tENtq0NDI4V28WF23qBgyLBpErQWIQE8J?=
 =?us-ascii?Q?/JTu5pCoXY7SpGnweQLoicv/ndzvf+tPrsJIoqZ84052xW9cFRPBhrMWpoFd?=
 =?us-ascii?Q?tHCar/dQ6/7shfZhBpxb/A6agsWJ1PEnrfAISLiIcvfDr7qeeayPEnvXqn/E?=
 =?us-ascii?Q?BqM6IttCiKZJ0zMju54JeZdd1ZfLsFLjoRTCzCmtzi8NtSvxmIkknCAFXlnN?=
 =?us-ascii?Q?NJNQwhy2pBmI2/pbSTLE7ymE/t+xHjPGxVYwUQXCcxSQuQL2zJl0fYEOZIj6?=
 =?us-ascii?Q?3fUs17IaG719SpdKTa+9TOjfbg21sQEDZxFamMaL5V1ehmxjABHew3Z9ERB8?=
 =?us-ascii?Q?kNLPGPMCR7NFCREHfMZh0vGyaQ0YfNrpNJaQPIu0Jm3hc1YoBW/Ja9Fp18Vz?=
 =?us-ascii?Q?l2N8WxkUdeiD6DG/DZF6faAhV/88zYQUhho7DPplTUmA4jSYHXbQHEMeirn0?=
 =?us-ascii?Q?CQAxrO4VCv03KUp/sZ7hAxv1fgJi6sw6PbA3ifwjfpwi4mToumeVQ0aZ6rkU?=
 =?us-ascii?Q?0bG1Nje8pJqNn5NttjSo3Shd9aYbCWfvbYYNCDX7koO14xG6M1JD3g5nzHFL?=
 =?us-ascii?Q?gMmdptkV7R5vrVG1rt159Mq+WQfifTW0pUwMVWKq0nXG9N1bft7WfpA7C4M4?=
 =?us-ascii?Q?oGKrSl8lKkcey7+pBtDFq58nDwE3jDOkrjxmW4GRn9xFlFPnSu6NMHd2paTM?=
 =?us-ascii?Q?Oz+gHflTCaGi+sINyRxjLqEbZFDGKU+ZhO6wbLiY7BRkfyy2LpXmEVJJrmeg?=
 =?us-ascii?Q?258Y9EUaHeTmamadkU2yiL6g/TnDV50xb+OfV+JKiR+jX/4TabFhhmi62pb2?=
 =?us-ascii?Q?D5zab4/yFWnWLXh6EP4/3zEvXJdMDPCVfvu0qUpjTjfW5wWklfwTIwfxVLaf?=
 =?us-ascii?Q?+SO4fcc/g/4qj2B2V7yP/8Wan+5GgjY7qhItekxXft6Nax9ha6U9RXEcvfcq?=
 =?us-ascii?Q?voprXlc5RseJ6wgjQ96BhanY?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5429.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a24f3286-a89e-4b6d-ac1e-08d9458e8d91
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jul 2021 23:41:21.1178
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: mQVt+VJYRJ/PlB45tFfNrmEwQc6Gb395/Y+1VuZ9P4m9MNfoYjumGuq8snQc7ocNOL9bpJ5rBynO3s1pwPateQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4109
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Tuesday, July 13, 2021 2:42 AM
>=20
> On Mon, 12 Jul 2021 01:22:11 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > > From: Alex Williamson <alex.williamson@redhat.com>
> > > Sent: Saturday, July 10, 2021 5:51 AM
> > > On Fri, 9 Jul 2021 07:48:44 +0000
> > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
>=20
> > > > For mdev the struct device should be the pointer to the parent devi=
ce.
> > >
> > > I don't get how iommu_register_device() differentiates an mdev from a
> > > pdev in this case.
> >
> > via device cookie.
>=20
>=20
> Let me re-add this section for more context:
>=20
> > 3. Sample structures and helper functions
> > --------------------------------------------------------
> >
> > Three helper functions are provided to support VFIO_BIND_IOMMU_FD:
> >
> > 	struct iommu_ctx *iommu_ctx_fdget(int fd);
> > 	struct iommu_dev *iommu_register_device(struct iommu_ctx *ctx,
> > 		struct device *device, u64 cookie);
> > 	int iommu_unregister_device(struct iommu_dev *dev);
> >
> > An iommu_ctx is created for each fd:
> >
> > 	struct iommu_ctx {
> > 		// a list of allocated IOASID data's
> > 		struct xarray		ioasid_xa;
> >
> > 		// a list of registered devices
> > 		struct xarray		dev_xa;
> > 	};
> >
> > Later some group-tracking fields will be also introduced to support
> > multi-devices group.
> >
> > Each registered device is represented by iommu_dev:
> >
> > 	struct iommu_dev {
> > 		struct iommu_ctx	*ctx;
> > 		// always be the physical device
> > 		struct device 		*device;
> > 		u64			cookie;
> > 		struct kref		kref;
> > 	};
> >
> > A successful binding establishes a security context for the bound
> > device and returns struct iommu_dev pointer to the caller. After this
> > point, the user is allowed to query device capabilities via IOMMU_
> > DEVICE_GET_INFO.
> >
> > For mdev the struct device should be the pointer to the parent device.
>=20
>=20
> So we'll have a VFIO_DEVICE_BIND_IOMMU_FD ioctl where the user
> provides
> the iommu_fd and a cookie.  vfio will use iommu_ctx_fdget() to get an
> iommu_ctx* for that iommu_fd, then we'll call iommu_register_device()
> using that iommu_ctx* we got from the iommu_fd, the cookie provided by
> the user, and for an mdev, the parent of the device the user owns
> (the device_fd on which this ioctl is called)...
>=20
> How does an arbitrary user provided cookie let you differentiate that
> the request is actually for an mdev versus the parent device itself?
>=20
> For instance, how can the IOMMU layer distinguish GVT-g (mdev) vs GVT-d
> (direct assignment) when both use the same struct device* and cookie is
> just a user provided value?  Still confused.  Thanks,
>=20

GVT-g is a special case here since it's purely software-emulated mdev=20
and reuse the default domain of the parent device. In this case IOASID
is treated as metadata for GVT-g device driver to conduct DMA isolation
in software. We won't install a new page table in the IOMMU just for
GVT-g mdev (this does reminds a missing flag in the attaching call to
indicate this requirement).

What you really care about is about SIOV mdev (with PASID-granular
DMA isolation in the IOMMU) and its parent. In this case mdev and
parent assignment are exclusive. When the parent is already assigned=20
to an user, it's not managed by the kernel anymore thus no mdev
per se. If mdev is created then it implies that the parent must be
managed by the kernel. In either case the user-provided cookie is
contained only within IOMMU fd. When calling IOMMU-API, it's
always about the routing information (RID, or RID+PASID) provided=20
in the attaching call.

Thanks
Kevin