Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Liu, Yi L" <yi.l.liu@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Thread-Topic: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Thread-Index: AQHXyZ9N9hHTiUakTUi0phVNj8QgQ6vpuU9ggATxeICAAVDBMA==
Date:   Tue, 2 Nov 2021 09:53:29 +0000
Message-ID: <PH0PR11MB565808A9C9974A0D0D72B738C38B9@PH0PR11MB5658.namprd11.prod.outlook.com>
References: <PH0PR11MB56583D477B3977D92C2C1ADDC3839@PH0PR11MB5658.namprd11.prod.outlook.com>
 <20211025125309.GT2744544@nvidia.com>
 <PH0PR11MB56586D2EC89F282C915AF18DC3879@PH0PR11MB5658.namprd11.prod.outlook.com>
 <20211101125013.GL2744544@nvidia.com>
In-Reply-To: <20211101125013.GL2744544@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-reaction: no-action
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Q6AFt0BZTEkZxOXA9ZCBzRfwk1FQiDe2MDWUj8YOrhDfHgj/6kPG0HjLE4se?=
 =?us-ascii?Q?yL8T0AhMQpSCOJlpjWXdyp448HJrv+28JPc7hSWExprlBa3yQZjjL8qmzBs5?=
 =?us-ascii?Q?5PQz5FUCaWxFOFhNWlpL9m24i1z1aeggxxVjUaGZNWm4OU3LhHWL2z3OO0RK?=
 =?us-ascii?Q?u2bBNUxEnOZDKGgnaUSfDdXUAIoAk4S2ep8ik8ykf4XUiaz3W4zwtqkx9i5a?=
 =?us-ascii?Q?CY4lNCXuLxumxEzHQjciAr3YWdAozEH4whF+JKI2jSIWPBzFpI22b2Eypcjj?=
 =?us-ascii?Q?1rh7xQ1U0MihSp1LpWwAhRHdgzXuKz36Rk0qh26N19chXniUKOidgyTHc5gj?=
 =?us-ascii?Q?BAcvu1d1LCBKw5vZPtxMjn2F0Ga3JLg3ryAWWPYR+wNJFTvim1TK1MYgvjTm?=
 =?us-ascii?Q?Xzb3WylrjqxJmdkm1Oj8O9yWccrDU9aH+cXJP+stjkda+vLOW0FLb2CT5UaQ?=
 =?us-ascii?Q?GR6PguZT6En4kH14IYc8A3lw2UsbmaWqQSQJ5OgeDilBIUMc2H0AmUNheZkU?=
 =?us-ascii?Q?in7mKb/JfAKMgoS8PP2TH9BUtJ0ek1KhzLpv2O0QFStkTlousfMi2ZJPkL+k?=
 =?us-ascii?Q?GPnIAQW9bHqYrxFyAnwpXe69G2AJkt02SsHe45sXSX5Z9hhxIZmAl8m47iPJ?=
 =?us-ascii?Q?giJ1VSLtQkRr7kFd2c8GEvQzBMtL7jzDEwF2azMqOJKZttckNAyfLaTpyD7S?=
 =?us-ascii?Q?9k+nRsUqubMgZAoY1cwhoRpkN1PqbyZzLb3IW3ir6+DT97xmkSng+4NODSY6?=
 =?us-ascii?Q?2AHiCmbDUzPB4NBt92QeLxuXlbTdK35lIwzhe8OX+1tGu4eUBRguNQ4gIeCp?=
 =?us-ascii?Q?WLwF9LP6WPj2A72IbMI735OEzRlyQJBdDylSHoRXEIbYAFaHpj00xFn/6jBM?=
 =?us-ascii?Q?K75SNcItPmYIXfKL4LkhA3sYjFn/y4WUd2ALtxKJUtIKNEzvsHepdCXFrx6L?=
 =?us-ascii?Q?OnG1VgXUU+0ThsDcR9scJbiRBj0PWy1pYC06PSTNY0edTJurQuY7uokch9xj?=
 =?us-ascii?Q?FvhRSH6N3NLMohHdr8Mq/R0+3S/tZdVgs6WD9CLwFFuYj2gAPDuGI0AtrMnJ?=
 =?us-ascii?Q?4XOuONhrcQ7kaVnuzbONO0ewjkpHjnHLhz0/VES7UCp6wxjq9oQfL60iwdIr?=
 =?us-ascii?Q?2/L+fnTxUkE1GGvSf0uYXl9vXAo/+8Mj+V9BgMI19y/RDj87fCeh3QvvmqiK?=
 =?us-ascii?Q?vDx1BkWL5pF/7LwK6CoZp47lJTQeuPMYyejSygoVWKB0FfeUca6BWcGTczNz?=
 =?us-ascii?Q?MmXvpgu9LL5CbwqS267OLCWdgdseSerDRpMFgSsC9Epml92ecKrA8QwuncM/?=
 =?us-ascii?Q?xH1PFX6wiDyK3U+Ue5biRSIw/2b+/tULiBu/uk6cw8yAXMN8ORCsfn0Fpl9X?=
 =?us-ascii?Q?i7YTvJa2eBuVOOC/sL9cSvr8B0d4OVaRu2Coddan3R9lgRRvFxLJS4Hk2eGy?=
 =?us-ascii?Q?a4covrjUKq8ntttOpob1OONfzNqA7PxN6uv89nfDU5brb6BJ7tOzOGArQdbc?=
 =?us-ascii?Q?e7WkTdSqVV5vOpMn3dHOVa2oT1vKFfGApHyLWeLHe4VEy0h8J0+Vyys1zJbt?=
 =?us-ascii?Q?BUrTPdLGHMMhrRu+9BcQ9oUM6mdB8M8uEcqryR66Ugf+sPtSYAmFRDUhXy46?=
 =?us-ascii?Q?vU9N6929ecycqhRqkERYzgY=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5658.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 14139b07-7b97-48f9-2b5b-08d99de6a007
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Nov 2021 09:53:30.1025
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: x8KZ3zYaPim3jmJPtK3sDpL3zW9OR2ZZvB+L0YujIahGRbEfwEMers1JJSIXcBTdgKo4PNSrDMqEGgphX7mChw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5660
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Monday, November 1, 2021 8:50 PM
>=20
> On Fri, Oct 29, 2021 at 09:47:27AM +0000, Liu, Yi L wrote:
> > Hi Jason,
> >
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Monday, October 25, 2021 8:53 PM
> > >
> > > On Mon, Oct 25, 2021 at 06:28:09AM +0000, Liu, Yi L wrote:
> > > >    thanks for the guiding. will also refer to your vfio_group_cdev =
series.
> > > >
> > > >    Need to double confirm here. Not quite following on the kfree. I=
s
> > > >    this kfree to free the vfio_device structure? But now the
> > > >    vfio_device pointer is provided by callers (e.g. vfio-pci). Do
> > > >    you want to let vfio core allocate the vfio_device struct and
> > > >    return the pointer to callers?
> > >
> > > There are several common patterns for this problem, two that would be
> > > suitable:
> > >
> > > - Require each driver to provide a release op inside vfio_device_ops
> > >   that does the kfree. Have the core provide a struct device release
> > >   op that calls this one. Keep the kalloc/kfree in the drivers
> >
> > this way sees to suit the existing vfio registration manner listed
> > below. right?
>=20
> Not really, most drivers are just doing kfree. The need for release
> comes if the drivers are doing more stuff.
>=20
> > But device drivers needs to do the kfree in the
> > newly added release op instead of doing it on their own (e.g.
> > doing kfree in remove).
>=20
> Yes
>=20
> > > struct ib_device *_ib_alloc_device(size_t size);
> > > #define ib_alloc_device(drv_struct, member)                          =
          \
> > >         container_of(_ib_alloc_device(sizeof(struct drv_struct) +    =
          \
> > >                                       BUILD_BUG_ON_ZERO(offsetof(    =
          \
> > >                                               struct drv_struct, memb=
er))),    \
> > >                      struct drv_struct, member)
> > >
> >
> > thanks for the example. If this way, still requires driver to provide
> > a release op inside vfio_device_ops. right?
>=20
> No, it would optional. It would contain the stuff the driver is doing
> before kfree()
>=20
> For instance mdev looks like the only driver that cares:
>=20
> 	vfio_uninit_group_dev(&mdev_state->vdev);
> 	kfree(mdev_state->pages);
> 	kfree(mdev_state->vconfig);
> 	kfree(mdev_state);
>=20
> pages/vconfig would logically be in a release function

I see. So the criteria is: the pointer fields pointing to a memory buffer
allocated by the device driver should be logically be free in a release
function. right? I can see there are such fields in struct vfio_pci_core_de=
vice
and mdev_state (both mbochs and mdpy). So we may go with your option #2.
Is it? otherwise, needs to add release callback for all the related drivers=
.

struct vfio_pci_core_device {
	struct vifo_device vdev;
...
	u8 *pci_config_map;
	u8 *vconfig;
...
};

struct mdev_state {
	struct vifo_device vdev;
...
	u8 *vconfig;
	struct page **pages;
...
};

> On the other hand ccw needs to rcu free the vfio_device, so that would
> have to be global overhead with this api design.

not quite get. why ccw is special here? could you elaborate?

Thanks,
Yi Liu