Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 21 Sep 2021 21:55:01 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 02/20] vfio: Add device class for /dev/vfio/devices
Message-ID: <20210922005501.GD327412@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-3-yi.l.liu@intel.com>
 <20210921155705.GN327412@nvidia.com>
 <BN9PR11MB5433A709ED352FD81DDBF5A68CA19@BN9PR11MB5433.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5433A709ED352FD81DDBF5A68CA19@BN9PR11MB5433.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kctm95fTwD6gdKbE7etdqLQmiq4kA5DPEB4LfBDUoLfx9oih35+2h9t4JThP?=
 =?us-ascii?Q?i7N5a7QjolafbbXomItxiKyJgiVKLJVfx+Rp33XmoarlyGCf54TAsV7Oiu2Y?=
 =?us-ascii?Q?a1Cv5uYgu1St/JfHpRa/7WbIc+OmDWrzmequ3TyR5HKp8IyzMEa5gMHpJtj6?=
 =?us-ascii?Q?Po+I0iwbMbWCbW9cQhd4VZO76mm2fJjDFEqsYjx7C2jOczUSa1db+i1k25zH?=
 =?us-ascii?Q?GIGEbqw7QT7b6f1LHra/VteA2Q6t2FF9//WyyXdqldHLH8sEj+Yh5nHUBXWX?=
 =?us-ascii?Q?usejXntXOzzy0Ogp374VHU49ALB6NbpBQOXpgu64nI5zqUbi678HmnjV4bnm?=
 =?us-ascii?Q?P7Lp5W/ogwlCOjJnpCqvByKae2bUXwERts+qY2QDZ+a/fnElGVKFf8NpUks3?=
 =?us-ascii?Q?qBqw3cc/wnsKJzf2pb9K5OVhkH3++gNn/Hm+J0IAxkSBvg0B+dEK8sXttc3q?=
 =?us-ascii?Q?46UbzRnzCevVw32m3SHrd4dRrpR3g1JTfB2lfvoYEqSqNT/LfoG5IrrjN+HW?=
 =?us-ascii?Q?X2giYbnZlaIawS6mAcrT/XjVc0Yob53nYY16fyQ2tgigYsyjp+3YR7GrPUW/?=
 =?us-ascii?Q?YyLFfrtpki8fEACsKMOOY1keuKxjoIdYxuv02C8NtjJxZeuZd5xHujxNsdm4?=
 =?us-ascii?Q?DsfUO0k60SwgoXLCzBxXdfMjowrv1r34oiIjX24jlIJuRDTMnN1XOu2WvYfr?=
 =?us-ascii?Q?BakktZic+CuvgFPpr8g0Kj86OGFzSurOfUjrx4AdcttzW23mWLzduQMV6UtZ?=
 =?us-ascii?Q?4ONb1+xuQIjAo5qIl8IQViOadxhMQT7q0HAnVpCrWgDjT+JYwK/V5EG3uuXA?=
 =?us-ascii?Q?xki/tKqXSrRCialraU9+6bW+OzZbJT55CvDnZg6s45KydIAXI6EYq+14yqEz?=
 =?us-ascii?Q?AP7rcaulR+Be0RP+xLhX5aTmRa7JfTyW76Z8ELAMIfhXx5nwH3UMLmSixA5z?=
 =?us-ascii?Q?u7nVDK1USs+kcNfwzNvxVSyKMnSWuKebcLUIdfYOpQ1INLdOnh4DS3QpU9Sa?=
 =?us-ascii?Q?JamSYSBLE/LSbOPCSsgpnl5X/1Vo4uZYsq7vVCmqsdKEtH9x51+z6lDQNUqS?=
 =?us-ascii?Q?ZCASeWzihaYqtjEhtp2DdX09IG56K8B445tUH05x8+tVAq48eR4b7fvGsrs/?=
 =?us-ascii?Q?prC/ypJ1ABs5IC4V+jZk+SVVrfoetU+7pe2dR8KFPOkb71xAJsI8+2Fw2iCD?=
 =?us-ascii?Q?P2C0Gt1AbmzzcDrIZBHUEvHOC37l94yjW4akmm723EAEASZ+CmmprdwMO9KL?=
 =?us-ascii?Q?B22JUnRcr62RBj73PGv6uYJAmrEaRHH69rTg766w62pT8sIKCBgL+j5TubPO?=
 =?us-ascii?Q?8pWiCIHEORqX1LoBbxENv76j?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 49fe2311-935a-4ba1-a050-08d97d639c02
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2021 00:55:02.4169
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iqAMaSmmk3iPf02w0jt3FL4qXGOCj7WQmwO2QnLAEeEPDA345PUaO0jeWVKSULPJ
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5173
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Sep 21, 2021 at 11:56:06PM +0000, Tian, Kevin wrote:
> > The opened atomic is aweful. A newly created fd should start in a
> > state where it has a disabled fops
> > 
> > The only thing the disabled fops can do is register the device to the
> > iommu fd. When successfully registered the device gets the normal fops.
> > 
> > The registration steps should be done under a normal lock inside the
> > vfio_device. If a vfio_device is already registered then further
> > registration should fail.
> > 
> > Getting the device fd via the group fd triggers the same sequence as
> > above.
> > 
> 
> Above works if the group interface is also connected to iommufd, i.e.
> making vfio type1 as a shim. In this case we can use the registration
> status as the exclusive switch. But if we keep vfio type1 separate as
> today, then a new atomic is still necessary. This all depends on how
> we want to deal with vfio type1 and iommufd, and possibly what's
> discussed here just adds another pound to the shim option...

No, it works the same either way, the group FD path is identical to
the normal FD path, it just triggers some of the state transitions
automatically internally instead of requiring external ioctls.

The device FDs starts disabled, an internal API binds it to the iommu
via open coding with the group API, and then the rest of the APIs can
be enabled. Same as today.

Jason