Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 1 Nov 2021 09:50:13 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Liu, Yi L" <yi.l.liu@intel.com>
Cc:     "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Message-ID: <20211101125013.GL2744544@nvidia.com>
References: <PH0PR11MB56583D477B3977D92C2C1ADDC3839@PH0PR11MB5658.namprd11.prod.outlook.com>
 <20211025125309.GT2744544@nvidia.com>
 <PH0PR11MB56586D2EC89F282C915AF18DC3879@PH0PR11MB5658.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <PH0PR11MB56586D2EC89F282C915AF18DC3879@PH0PR11MB5658.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?rz1sbf37qcHz5M/N1BDUI+Zr2dpe+HEbc9Ho7HGj24WOAH8N3DSopRgVBlei?=
 =?us-ascii?Q?RqH/OiQ3MNPEcO2jkm8UOQGXcLY8RyejdZy9hNoawhVZ1IHAIz7gQ4AaY9sY?=
 =?us-ascii?Q?QrlNbirb0hxZLpD6AVgUoQGkzpE70f4wzNdprSsT9IS5ogQHR8hoHMpX/Afp?=
 =?us-ascii?Q?aSDXwEdinkGr93nSwFX5M8gOD7QTzo0n6bXKXebQ0/xo/zKtVEmS7Rc+Ni9h?=
 =?us-ascii?Q?ZHfvqK5bznM0eZBInWMlXiCneHLaEE8FtofYyukV0UShFrsSLfWFfnfF+har?=
 =?us-ascii?Q?1RLA8FJ+EzXbtXSehwW4nRLyRI4cfZzYnFCKRQqS1y8LA9m3D19hIpT4TBWN?=
 =?us-ascii?Q?R2BsfpYoL546TK4C+JcsX/ggjfFQASUWJgSYplGQhc1v8dCgz34tOpym5dVq?=
 =?us-ascii?Q?VJSf93k44fC4pxA+QW1THAMEV3r5thne7/VLiP1LwzPRpdtRgTpS6jRiJfDx?=
 =?us-ascii?Q?PfxCX7JFpYKqUJHla50iqTvHbmqOZth9Qql9b9+wDZ/ztTge3JSUSXNgHY/p?=
 =?us-ascii?Q?549PG+WECzq0Ew1QalXqovweK9NhQi6Hfbxr/lVDkc/dil/J+XLsLGO6RAUq?=
 =?us-ascii?Q?LfuHRBC78YDAPJ8m1PDWHixTEBzhub1VXE6yqfqgNTjw/u09EfBN9LsICEEH?=
 =?us-ascii?Q?KbAY3rv8+SwWoQS+ALt3sb8qZ8TDWmQSCnbN90mkpAHlFaAWJjW4Owl7uhlk?=
 =?us-ascii?Q?FsjUPWa8tPhfjWaLFSKyrauO9HEw1FLm7gWnKr29rIashcMU4reNn05DGrcm?=
 =?us-ascii?Q?KA2IRUAd55hfTwnhWkjVB5T/VeRKcG3bUbIYLMMNz76f+jdUFmttaPXp8/DL?=
 =?us-ascii?Q?BVspS15TzmkK5kBh+57008yVVVVg88hu9xTXFpfWQ0TK1yMfE/cuoULLltBG?=
 =?us-ascii?Q?RwG94ux/xy0KxX3c5uvWNq72B0QU3LVmsku6Ak9WrlsX4Gf7xXT3Timm+UHp?=
 =?us-ascii?Q?RRigsENLpHKnnLhMlNOgoS3QB2W3a1Y5SZrjzuO/o+rJRmfCNe8TAeDjGbM6?=
 =?us-ascii?Q?RlN3dHwP4RsHyug8TPo92Y3Yq6sgHM9Cjwq9QhmsDMKx9yMFq6R0UVH3icUx?=
 =?us-ascii?Q?S8SpCkHtejXz5SCzRl/BKmTZNUMO5sJ9eocfOg/K7bKUfUIw3hf4fMcskHvH?=
 =?us-ascii?Q?8aOPOwIst+IcTPMBFEzwWbbqXrWEusV11MFzMmWvqMowJiUy4eMFc4sCKVUa?=
 =?us-ascii?Q?ijJyGIYZlaNNXNH58Pw3f8+EymBuW0TwWRU6KD3VU46KhSBo+SqXrJW6f9fx?=
 =?us-ascii?Q?T6KeeeKXYdX25pLxZkjIXcpWr5dv6WQe+46CemKqS4kXuA6mZoxF1x+LONhW?=
 =?us-ascii?Q?AlnROK3lXlGPgu7Y6cOjOZKeORA+vpUQtNCCaZ7MjbmAJ6GXktu+r9hSPJPx?=
 =?us-ascii?Q?m21XP1nds7W7PBqyLQZuC2rxBVHIvk+vH12q7YykgU0uBkb9OiLkH6f7NCqb?=
 =?us-ascii?Q?J8dai7zoz64jOg8p4bge80KCIDqrQ0kCCAt0YVVX+1iGG68Hd8bE6+mipYUJ?=
 =?us-ascii?Q?O7o7hlk1+5vg39qvXJUKkZqPpKxANMv1ri7Rm1qPRHvtwwOGfGDFPemlFS7E?=
 =?us-ascii?Q?pK2f4/gf0HBq+F9P2h8=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bef42638-0698-4caf-8fb7-08d99d362617
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2021 12:50:14.5436
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3sHB65Cw47XuY6hf5aTGEWdqKp4UhLFi9OiPqOmL+cPtOaFtfSURHKOOt2t1jLvx
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5127
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Oct 29, 2021 at 09:47:27AM +0000, Liu, Yi L wrote:
> Hi Jason,
> 
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Monday, October 25, 2021 8:53 PM
> > 
> > On Mon, Oct 25, 2021 at 06:28:09AM +0000, Liu, Yi L wrote:
> > >    thanks for the guiding. will also refer to your vfio_group_cdev series.
> > >
> > >    Need to double confirm here. Not quite following on the kfree. Is
> > >    this kfree to free the vfio_device structure? But now the
> > >    vfio_device pointer is provided by callers (e.g. vfio-pci). Do
> > >    you want to let vfio core allocate the vfio_device struct and
> > >    return the pointer to callers?
> > 
> > There are several common patterns for this problem, two that would be
> > suitable:
> > 
> > - Require each driver to provide a release op inside vfio_device_ops
> >   that does the kfree. Have the core provide a struct device release
> >   op that calls this one. Keep the kalloc/kfree in the drivers
> 
> this way sees to suit the existing vfio registration manner listed
> below. right? 

Not really, most drivers are just doing kfree. The need for release
comes if the drivers are doing more stuff.

> But device drivers needs to do the kfree in the
> newly added release op instead of doing it on their own (e.g.
> doing kfree in remove).

Yes

> > struct ib_device *_ib_alloc_device(size_t size);
> > #define ib_alloc_device(drv_struct, member)                                    \
> >         container_of(_ib_alloc_device(sizeof(struct drv_struct) +              \
> >                                       BUILD_BUG_ON_ZERO(offsetof(              \
> >                                               struct drv_struct, member))),    \
> >                      struct drv_struct, member)
> > 
> 
> thanks for the example. If this way, still requires driver to provide
> a release op inside vfio_device_ops. right?

No, it would optional. It would contain the stuff the driver is doing
before kfree()

For instance mdev looks like the only driver that cares:

	vfio_uninit_group_dev(&mdev_state->vdev);
	kfree(mdev_state->pages);
	kfree(mdev_state->vconfig);
	kfree(mdev_state);

pages/vconfig would logically be in a release function

On the other hand ccw needs to rcu free the vfio_device, so that would
have to be global overhead with this api design.

Jason