Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Liu, Yi L" <yi.l.liu@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Thread-Topic: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Thread-Index: AQHXyZ9N9hHTiUakTUi0phVNj8QgQ6vpuU9g
Date:   Fri, 29 Oct 2021 09:47:27 +0000
Message-ID: <PH0PR11MB56586D2EC89F282C915AF18DC3879@PH0PR11MB5658.namprd11.prod.outlook.com>
References: <PH0PR11MB56583D477B3977D92C2C1ADDC3839@PH0PR11MB5658.namprd11.prod.outlook.com>
 <20211025125309.GT2744544@nvidia.com>
In-Reply-To: <20211025125309.GT2744544@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-reaction: no-action
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?VHACYHR3sj8UWMRZHBiKnP+hiydylG9fcANQaFMiIJNrpj41rLpVI7FKNAtN?=
 =?us-ascii?Q?ayy3fzgQ8R9Xhg4IObyi3Z33mPuQnzlrIObsSDbW8acN3k64CptsQBPImi44?=
 =?us-ascii?Q?JNgeSonI3oZmwLzizDncUkKgDAV6llwRET/5zba8163HW1zLZMRIiCrc9NBn?=
 =?us-ascii?Q?ui3ogs2N0W4ixF4fUlsnPqKpqjtOlzAdm6fuWS86P4o6ESQ4h5yPBSfHDpos?=
 =?us-ascii?Q?wvNBDewaaY3HGySeYjOY3c9r/FS4Luo2IBHWYtHl9oAoAax8/7x0vyCHbRNV?=
 =?us-ascii?Q?4wtNrNiHSs3QSOKNIrignKZyr0CqOva3bVgEHx99u8zF7i/VaUckvWOw2tNl?=
 =?us-ascii?Q?7euKs7WW9auQU2Hziob5g/HUh2aXlEWjVgMYqfT0XFUY2EqdCsZh5rOW34L7?=
 =?us-ascii?Q?vsVv4XGzcu7HLiRtbCKjGtG5bzEKdQza4vRj4kQavCPeI7qdJogOcoIpOm9P?=
 =?us-ascii?Q?8fQPeY7ONv6E+le0vhPqxxFRi3tsf3WzujjekA2W2+oXHVK2Pk9TQ8hbe5VB?=
 =?us-ascii?Q?q3iEgog4m2W53xdrzdmdLDut46WxEt99BFflZrzYseKaxmc8nnzBRso3FmV1?=
 =?us-ascii?Q?ZIjl1zOS/43D9o9L8HWdlH4D3Wg1m05ufdunSlpjJqr+C1yVM7nvbYjUyaek?=
 =?us-ascii?Q?7ctghH4uPwmaz2qzsbcAL+cov9DlK7Uz1+dHIHuMMyQ9VNXDh1PpJuig/cVO?=
 =?us-ascii?Q?LsdmPT4PchDGNxyJiwS2xiqtqUbAqTQkMTZgudNJpZrtMpZQUJ60oVcM9JUZ?=
 =?us-ascii?Q?2gBjJloRXDyVyofYMAF6qx0a65R8DWIkPyagRWrcaQy8u7jR2HGueOjxhT60?=
 =?us-ascii?Q?oKAZcM7vP0Zepgye+IDiiukIjMl4W2d6WCCcuaxDYVu3kgZHJYQPJwHfczRN?=
 =?us-ascii?Q?I9vBekQX/juWHBgIigfystYMQPDHQkmT8h8BziOUXl7v7PkxIfoG+pz4SQ5U?=
 =?us-ascii?Q?oKaXqYiDnxcEeIfvAp2mX82hyaoIbw1MlH5hjm1vjlmPOSCbWkG7UZBj1kKI?=
 =?us-ascii?Q?z6enztUH4mu2mUCjkz+za4IJuNtbo3OHG+rUqcVe/cXZvGH7t7bREaRg4iHp?=
 =?us-ascii?Q?m6WqkBn4R0MwOesp5EH+VVhFr/zQAYQPfUGtMCktxZzKJUqK/d9e24nbZ0SP?=
 =?us-ascii?Q?qBjfi7RHzAPLsTOU/JvX3ewH3LjYtbzgqyOKFdrxToCrpCkCvujCjwrfWoUd?=
 =?us-ascii?Q?rKGTvClwCZwbIVk7b5xUiQCHjBCIFOmsV4aXLHq9KuJVKwrorhXhxc7AGFvd?=
 =?us-ascii?Q?7PwSmWGlrXTo3GFBmpWU5usnPKXaoITUT3ZVjUROjS8H/U35cIvU8cpZWhLD?=
 =?us-ascii?Q?mARxt9QJGtDmIBqsvXoTCGUFbRpGWCacPL3YQcRrpM+KmLsQarDUsmBPk389?=
 =?us-ascii?Q?fHVlkwHvrLiHGiUmE3F8SsawzYb+psLwD/9+35CbAE2kUSySPoUko7t9/Zys?=
 =?us-ascii?Q?EP7Q1oLh+7QrP9Hy1HHcLPhbYXIOt24sph+Yr4QJ8pXkJMO15dA4m8fa+wlA?=
 =?us-ascii?Q?Z/DqhX0Lg8CeoPzY+3hX467mYnKRhWFrlq0ENO7fGAm4nUq/WaNKWrb1cc22?=
 =?us-ascii?Q?Uix2d4M0yuhLWU27HxgDAZXfigTwnk6ctvuInrA0ni/IRiQJdV+w4a1qNZ7I?=
 =?us-ascii?Q?wCT1lXkIE5/zshGWySVBshY=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5658.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 80c5d844-cd0e-475b-9036-08d99ac11e2e
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2021 09:47:27.2641
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Ks9I9MXQReUkj/WbTPohaU8T5RJR8S1hrsX++E/f5jvyzZS2X0ieGUrL9WMwYY00PJR5Tddd8E98Bd9+x5FGHA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5644
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jason,

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Monday, October 25, 2021 8:53 PM
>=20
> On Mon, Oct 25, 2021 at 06:28:09AM +0000, Liu, Yi L wrote:
> >    thanks for the guiding. will also refer to your vfio_group_cdev seri=
es.
> >
> >    Need to double confirm here. Not quite following on the kfree. Is
> >    this kfree to free the vfio_device structure? But now the
> >    vfio_device pointer is provided by callers (e.g. vfio-pci). Do
> >    you want to let vfio core allocate the vfio_device struct and
> >    return the pointer to callers?
>=20
> There are several common patterns for this problem, two that would be
> suitable:
>=20
> - Require each driver to provide a release op inside vfio_device_ops
>   that does the kfree. Have the core provide a struct device release
>   op that calls this one. Keep the kalloc/kfree in the drivers

this way sees to suit the existing vfio registration manner listed
below. right? But device drivers needs to do the kfree in the
newly added release op instead of doing it on their own (e.g.
doing kfree in remove).

vfio_init_group_dev()
vfio_register_group_dev()
vfio_unregister_group_dev()
vfio_uninit_group_dev()

> - Move the kalloc into the core and have the core provide the kfree
>   with an optional release callback for anydriver specific cleanup
>=20
>   This requires some macro to make the memory layout work. RDMA has
>   a version of this:
>=20
> struct ib_device *_ib_alloc_device(size_t size);
> #define ib_alloc_device(drv_struct, member)                              =
      \
>         container_of(_ib_alloc_device(sizeof(struct drv_struct) +        =
      \
>                                       BUILD_BUG_ON_ZERO(offsetof(        =
      \
>                                               struct drv_struct, member))=
),    \
>                      struct drv_struct, member)
>=20

thanks for the example. If this way, still requires driver to provide
a release op inside vfio_device_ops. right?

> In part the choice is how many drivers require a release callback
> anyhow, if they all do then the first is easier to understand. If only
> few or none do then the latter is less code in drivers, and never
> exposes the driver to the tricky transition from alloc to refcount
> cleanup.

I'm not quite sure. But per my understanding, since the vfio_device
is expected to be embedded in the device state struct (e.g.
vfio_pci_core_device), I guess most of the drivers will require callback
to do driver specific cleanup. Seems like option #1 may make sense?

Regards,
Yi Liu