Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Alex Williamson <alex.williamson@redhat.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 05/20] vfio/pci: Register device to /dev/vfio/devices
Thread-Topic: [RFC 05/20] vfio/pci: Register device to /dev/vfio/devices
Thread-Index: AQHXrSF8PdwE3l0WRkaTif5FzzsyqKuutGGAgABLSoCAAA3FAIAAOBXg
Date:   Wed, 22 Sep 2021 01:24:57 +0000
Message-ID: <BN9PR11MB5433929A6B9F8A2E35612D6F8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-6-yi.l.liu@intel.com>
 <20210921164001.GR327412@nvidia.com>
 <20210921150929.5977702c.alex.williamson@redhat.com>
 <20210921215846.GB327412@nvidia.com>
In-Reply-To: <20210921215846.GB327412@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FgI6fktnT40Zy1RKqZS7Np2pXh442kMyMtm5BHImxqOagjIOB80dLFtAIJ0t?=
 =?us-ascii?Q?mqbiuBXYIYaMW4CJgM0ujL+4pNQeq2RZQwnuj4v85YS6IK4Yw1B86uHNpgSX?=
 =?us-ascii?Q?+KgZjMzd2nRYwBkPvzkaruRMlSmtyJft8CBUERNdqZcS/M+hNNTuvAUymceJ?=
 =?us-ascii?Q?JO0TwQbFpdLeA3ozW9IExPSeuG5NIEKdP4OLDdBZLeqDbc5ELQbmZjJfiFVY?=
 =?us-ascii?Q?IJsM8LTfd5O1v1gUjqCnM3geU+fGlSjOxmE+prIaDIbjUPL0jOzxKBi5KOUg?=
 =?us-ascii?Q?LdezFw9n6RFAt/LdInVbkGMEg4LItM16JHyOAM4EUtc8Iv78CJ3qFfTntLN4?=
 =?us-ascii?Q?fN3elqMabschdyZkMezDmzlSsE3HK72+5dfesU2bzf8KoLItBj4GJSJtRaUB?=
 =?us-ascii?Q?pJ/EFuFo0tNgN23X5tsvaeTMnrRrmMxCRt6icL59ouIA2KE3lj5obHbuZpzl?=
 =?us-ascii?Q?lIFA8l1XRgekEJlQxwE1f5pOV5/7fwZLjCXwZFoQ+FC7oouzjd84/zkUiTBx?=
 =?us-ascii?Q?Y+bG+OGBOxIIvCo9D4kiCkHI5eVBfKReXBUbAFW7xK5E50tJpn559hJj1z+S?=
 =?us-ascii?Q?1gCQ6RpZChTd/mf0OurlUypMx5k9x8yfO9uTH59LRPU5M4t/gZuHSwMpqYrI?=
 =?us-ascii?Q?vajPm8fbmFgQpI/HDB4oaxBXYyu9etFHc/pW6KMYKYDNCPAJBx28qThplAP2?=
 =?us-ascii?Q?QRTGLeWI2RQDJul+rdHcOkBurow/TI/xAu0D/RtTeMgGpVaQUDnQjo5uFu49?=
 =?us-ascii?Q?Q/o1h8esHfHUbqSJZd/WWZ3AVBN/ta1XGXYe4VzAfwCVMpVriy+0zYngKZFG?=
 =?us-ascii?Q?3a69nKtWrX+LejBrWnuueITpoXBDqsIKFi8TRkoGsdpXmm77roW1IMO7JwjO?=
 =?us-ascii?Q?qAqgBBNDegATLdTbwRUhYkzIwbLXBO0jUgq6Q/KueEejeEuspWk0teWT85tl?=
 =?us-ascii?Q?Mh4KKzVhY6WNCQMZOYB0OTuZJgOffUSHH8MSzAzY2DG7oCxERQi7Qy2STwba?=
 =?us-ascii?Q?wMLsSubvClKD9FqKLKTCIDzwaUD+ppKnv4FvDubAalg5bqNscwxyxJOS+DO5?=
 =?us-ascii?Q?9YCRtCuFRrwmDTooy/F0oT0Z6CgJs7NjyJhGJ2eKkXj1KeOPvMx183dC3+EZ?=
 =?us-ascii?Q?Gtkm2DGqptzS4FeXOBwzJ0ecz0kmQxg59XIz0lrWDvS+/EAU8YWYl+XaNQPK?=
 =?us-ascii?Q?uBGhSiYolOHSDOYTRKb28oEuJ0RcH/luraWknBhl+aT+tHbd03PZkYLNh8Iy?=
 =?us-ascii?Q?s88Yuo3a4yqNwYNKw/vsR8a0rIyStejhTVG+plgtfEFGVKWCx3XfSLEM5VSg?=
 =?us-ascii?Q?07O1X3tHoTXjY21uYk2/TKad?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cf2872c2-2d9e-4dbe-be75-08d97d67c9fd
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Sep 2021 01:24:57.2090
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: VNh6xgPMIS26cgBysgBYDejwdmf1u7qW8/9PVTOd0Jdn9cEkEpQLDg8JKoR5yRFPPfyKimc4VzAfo+eZQZrvrw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1329
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, September 22, 2021 5:59 AM
>=20
> On Tue, Sep 21, 2021 at 03:09:29PM -0600, Alex Williamson wrote:
>=20
> > the iommufd uAPI at all.  Isn't part of that work to understand how KVM
> > will be told about non-coherent devices rather than "meh, skip it in th=
e
> > kernel"?  Also let's not forget that vfio is not only for KVM.
>=20
> vfio is not only for KVM, but AFIACT the wbinv stuff is only for
> KVM... But yes, I agree this should be sorted out at this stage

If such devices are even not exposed in the new hierarchy at this stage,
suppose sorting it out later should be fine?

>=20
> > > > When the device is opened via /dev/vfio/devices, vfio-pci should
> prevent
> > > > the user from accessing the assigned device because the device is s=
till
> > > > attached to the default domain which may allow user-initiated DMAs =
to
> > > > touch arbitrary place. The user access must be blocked until the de=
vice
> > > > is later bound to an iommufd (see patch 08). The binding acts as th=
e
> > > > contract for putting the device in a security context which ensures=
 user-
> > > > initiated DMAs via this device cannot harm the rest of the system.
> > > >
> > > > This patch introduces a vdev->block_access flag for this purpose. I=
t's set
> > > > when the device is opened via /dev/vfio/devices and cleared after
> binding
> > > > to iommufd succeeds. mmap and r/w handlers check this flag to decid=
e
> whether
> > > > user access should be blocked or not.
> > >
> > > This should not be in vfio_pci.
> > >
> > > AFAIK there is no condition where a vfio driver can work without bein=
g
> > > connected to some kind of iommu back end, so the core code should
> > > handle this interlock globally. A vfio driver's ops should not be
> > > callable until the iommu is connected.
> > >
> > > The only vfio_pci patch in this series should be adding a new callbac=
k
> > > op to take in an iommufd and register the pci_device as a iommufd
> > > device.
> >
> > Couldn't the same argument be made that registering a $bus device as an
> > iommufd device is a common interface that shouldn't be the
> > responsibility of the vfio device driver?
>=20
> The driver needs enough involvment to signal what kind of IOMMU
> connection it wants, eg attaching to a physical device will use the
> iofd_attach_device() path, but attaching to a SW page table should use
> a different API call. PASID should also be different.

Exactly

>=20
> Possibly a good arrangement is to have the core provide some generic
> ioctl ops functions 'vfio_all_device_iommufd_bind' that everything
> except mdev drivers can use so the code is all duplicated.

Could this be an future enhancement when we have multiple device
types supporting iommufd?

>=20
> > non-group device anything more than a reservation of that device if
> > access is withheld until iommu isolation?  I also don't really want to
> > predict how ioctls might evolve to guess whether only blocking .read,
> > .write, and .mmap callbacks are sufficient.  Thanks,
>=20
> This is why I said the entire fops should be blocked in a dummy fops
> so the core code the vfio_device FD parked and userspace unable to
> access the ops until device attachment and thus IOMMU ioslation is
> completed.
>=20
> Simple and easy to reason about, a parked FD is very similar to a
> closed FD.
>=20

This rationale makes sense. Just the open how to handle exclusive
open between group and nongroup interfaces still needs some
more clarification here, especially about what a parked FD means
for the group interface (where parking is unnecessary since the=20
security context is already established before the device is opened)

Thanks
Kevin