Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 21 Sep 2021 12:57:05 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Liu Yi L <yi.l.liu@intel.com>
Cc:     alex.williamson@redhat.com, hch@lst.de, jasowang@redhat.com,
        joro@8bytes.org, jean-philippe@linaro.org, kevin.tian@intel.com,
        parav@mellanox.com, lkml@metux.net, pbonzini@redhat.com,
        lushenming@huawei.com, eric.auger@redhat.com, corbet@lwn.net,
        ashok.raj@intel.com, yi.l.liu@linux.intel.com,
        jun.j.tian@intel.com, hao.wu@intel.com, dave.jiang@intel.com,
        jacob.jun.pan@linux.intel.com, kwankhede@nvidia.com,
        robin.murphy@arm.com, kvm@vger.kernel.org,
        iommu@lists.linux-foundation.org, dwmw2@infradead.org,
        linux-kernel@vger.kernel.org, baolu.lu@linux.intel.com,
        david@gibson.dropbear.id.au, nicolinc@nvidia.com
Subject: Re: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Message-ID: <20210921155705.GN327412@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-3-yi.l.liu@intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210919063848.1476776-3-yi.l.liu@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gmti37FuoC2EKgRQnxpQlAYYuqDRA6azyU7oowJr1IZ3d6zGYt83LhqWQR+F?=
 =?us-ascii?Q?M3Ohfg5Bev2kdB9uncMbhAdfkIWJjjnFZUHmrm3dQbaDZDrwCKYOi9NbQCnl?=
 =?us-ascii?Q?16eCUNzf2dPFH5MsoAmED/qxZS3gYnC0V0SX6bf042Og5h3jtc0twmDAFcSO?=
 =?us-ascii?Q?wXbdRwINtILG5XeOvO80MHjweo5ePq46jWoe3q2rzoXeDSvvmouaWGCPFLGK?=
 =?us-ascii?Q?/K+s1VAAg+E+enfGiEGbdsnf86qvncrSNWDjEChoKlFGrG9Snyuwv7kVkuGM?=
 =?us-ascii?Q?0ppvRxs1J6teq8eagb+YTdLinIkZeE/LgMEUcueT0Bl+dv5Of0218Pv4W86N?=
 =?us-ascii?Q?QV/aD2jb99DdJX0trs+Zcl2oaSH2DPnhq8uwQ31iDpO5stxhb/rH+5ro2XDk?=
 =?us-ascii?Q?hXRrqm36ZwWDH9qe2Zgpayg30C4Wpj+kQyORHiw5W9o/cn+83g2+Qty0mfiA?=
 =?us-ascii?Q?G48W8+QF8k5dkDgnOz+fqqxVQR/I3OnyGnchPU+362T3PgyRnjd7yK2geo+9?=
 =?us-ascii?Q?bsxwT2eK6P/ai2A0+QpH3rRAf0xljvQtzdF7+PzyJkeH4LClX94eTQ5aqE3T?=
 =?us-ascii?Q?vkQVfToTLNalQ03XAhg4xgZOd9aeyUS4SvbfU4MmYqFWF9uZV6p3Geaa17SQ?=
 =?us-ascii?Q?qwF+7od8icMUdqA6dvbGnYopthGnAYXVf0EXMSn3IV1FixLBvyO480o2jYvA?=
 =?us-ascii?Q?z9yzyJaO3PoonCwgyRHl6rpnegV46haRXZcgmKsFjXyL53tCegmYC5fxbU3i?=
 =?us-ascii?Q?GQZeiBufVdmEBh8RcqmQC95C36fELyNSBX0PG5ieYAkRmDsJ4zNBoSLgGJHJ?=
 =?us-ascii?Q?dAr372M1OmJZVXQImcwsRvjUvPzvZDm9V8YIALtxW61/FVnReCLjvvbohrkv?=
 =?us-ascii?Q?sYyAivdut97K0IwVt9dGrxUHMUBa80+CqgeXLSXCER/pswUbUFFja+X6m09H?=
 =?us-ascii?Q?1GSsY8cNPwL5/bhfPqOODe+jkwnJ0ELCPfD9EG8kqulA2mISLuyI6e0xAlcL?=
 =?us-ascii?Q?MR5vCgSCmZjcGa1NiMQgr1aLV122JXq5OB2InHiDnJ53oX+Wb3l90i87PPYV?=
 =?us-ascii?Q?UsIM7DMjZJ0fIDD0q8duc0aTqjVdsnydGs1z9N7Ka8uFXq73WeLTNjY4Onxa?=
 =?us-ascii?Q?Y3djJ7Foqyh1rbsrzbYVtsLcHZByA/dE06xWMmeFqhqHmLao9Gn8eG3+j9Qm?=
 =?us-ascii?Q?OhjLHir1oZZpPf44iR+eVbVqojGWlyTfxgXk2Gpb5M40GcZXMDGj4OOMCq06?=
 =?us-ascii?Q?lafQ3xNn07HVF8QYCiLqTndqAJ6f2R07sRzYTy0n91EzD+ORynbTq9C6N89a?=
 =?us-ascii?Q?soVdof+Z371VHxY3Qsf8+zQG?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e199b9e9-4afd-4377-7241-08d97d1875ed
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2021 15:57:06.4960
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mcZdEx7FzhclTIUnSzh2W1CRzugQKfqxsOK2sryBbm4QseiuHQfPa8No9w9V5IBd
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5157
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sun, Sep 19, 2021 at 02:38:30PM +0800, Liu Yi L wrote:
> This patch introduces a new interface (/dev/vfio/devices/$DEVICE) for
> userspace to directly open a vfio device w/o relying on container/group
> (/dev/vfio/$GROUP). Anything related to group is now hidden behind
> iommufd (more specifically in iommu core by this RFC) in a device-centric
> manner.
> 
> In case a device is exposed in both legacy and new interfaces (see next
> patch for how to decide it), this patch also ensures that when the device
> is already opened via one interface then the other one must be blocked.
> 
> Signed-off-by: Liu Yi L <yi.l.liu@intel.com>
> ---
>  drivers/vfio/vfio.c  | 228 +++++++++++++++++++++++++++++++++++++++----
>  include/linux/vfio.h |   2 +
>  2 files changed, 213 insertions(+), 17 deletions(-)

> +static int vfio_init_device_class(void)
> +{
> +	int ret;
> +
> +	mutex_init(&vfio.device_lock);
> +	idr_init(&vfio.device_idr);
> +
> +	/* /dev/vfio/devices/$DEVICE */
> +	vfio.device_class = class_create(THIS_MODULE, "vfio-device");
> +	if (IS_ERR(vfio.device_class))
> +		return PTR_ERR(vfio.device_class);
> +
> +	vfio.device_class->devnode = vfio_device_devnode;
> +
> +	ret = alloc_chrdev_region(&vfio.device_devt, 0, MINORMASK + 1, "vfio-device");
> +	if (ret)
> +		goto err_alloc_chrdev;
> +
> +	cdev_init(&vfio.device_cdev, &vfio_device_fops);
> +	ret = cdev_add(&vfio.device_cdev, vfio.device_devt, MINORMASK + 1);
> +	if (ret)
> +		goto err_cdev_add;

Huh? This is not how cdevs are used. This patch needs rewriting.

The struct vfio_device should gain a 'struct device' and 'struct cdev'
as non-pointer members

vfio register path should end up doing cdev_device_add() for each
vfio_device

vfio_unregister path should do cdev_device_del()

No idr should be needed, an ida is used to allocate minor numbers

The struct device release function should trigger a kfree which
requires some reworking of the callers

vfio_init_group_dev() should do a device_initialize()
vfio_uninit_group_dev() should do a device_put()

The opened atomic is aweful. A newly created fd should start in a
state where it has a disabled fops

The only thing the disabled fops can do is register the device to the
iommu fd. When successfully registered the device gets the normal fops.

The registration steps should be done under a normal lock inside the
vfio_device. If a vfio_device is already registered then further
registration should fail.

Getting the device fd via the group fd triggers the same sequence as
above.

Jason