From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>
CC:     "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "intel-gvt-dev@lists.freedesktop.org" 
        <intel-gvt-dev@lists.freedesktop.org>,
        "intel-gfx@lists.freedesktop.org" <intel-gfx@lists.freedesktop.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>
Subject: RE: [PATCH v2 07/14] vfio: Block device access via device fd until
 device is opened
Thread-Topic: [PATCH v2 07/14] vfio: Block device access via device fd until
 device is opened
Thread-Index: AQHZOgo/RJPtWsqmDkunQHE1j0WQCq7C2hUg
Date:   Tue, 7 Feb 2023 03:52:27 +0000
Message-ID: <BN9PR11MB5276373EFEBE732B431D87408CDB9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230206090532.95598-1-yi.l.liu@intel.com>
 <20230206090532.95598-8-yi.l.liu@intel.com>
In-Reply-To: <20230206090532.95598-8-yi.l.liu@intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?WhJx9GP66nAo3TAdkT+6nY9Ioo9pZ0dxBV0EM6xhOLNsLaePFjrmhC9zIBCZ?=
 =?us-ascii?Q?o+4ejz6vDrPu2gyzPvfyN/gT20waW2Jvv2enYiiPYjyWahBcbeXZ5jtIaKd6?=
 =?us-ascii?Q?r8XfBj51xxIXKj5voMJCQI2zDhelRBSKvE3nY86tt6xX06nntjeQYyan8KTd?=
 =?us-ascii?Q?cLpl8fFp9kklMZehJz79LzRO/pwjAl4kaT1J6208yBJNzGmoYIvUbifh/iqV?=
 =?us-ascii?Q?OZRBssQlwjPz926z4lnI0996Vg64IMMw46+JolZhaKFNHFE7d7q4UN8+/lQv?=
 =?us-ascii?Q?xdp+hlCaHY5fFQkQRhgOXYyLSPA87M2eKis+56ew+XUuvcRSOBCMotvC/NO5?=
 =?us-ascii?Q?kQK4rke9YQ1fhV/Pi/QPbjK9faavueviNJQNYZLPMfTECgcmR6vb4OE0zdbj?=
 =?us-ascii?Q?m0a9EJKoLGI4mwho3y+1Wvxlk1Fg4XWsJnL9AjUqGY79TyNOtiLwS6JgblX5?=
 =?us-ascii?Q?SCBBc9cGFaDtoywjqPrkAmm3lKQhB0ULeGzM63Av6FSyeIJ85GkyvSuX7hDR?=
 =?us-ascii?Q?yjqL4OqsCU7+BS1Er8x9rVLcGaStHgmYlIpO53SvVjidpQ5BoCcMU3xMIYE6?=
 =?us-ascii?Q?kLCAnDgLA68RFXDdbAn6AeIeDI62Bc31m46PPVFTkSKC9CYfaWP5t8+le495?=
 =?us-ascii?Q?cDAEHxpY3aFFd18MeSxZVBug20sXi8W3ZSnAvq+EEPCHqiHiUlTIGeGO7c+l?=
 =?us-ascii?Q?OQo5brKeBYBxk2RJQOePBZh6l9qKp9BmfNDkaGlPhToDv8c9aWS5iixOOkR3?=
 =?us-ascii?Q?dCci4n2xyg2oK/K4tyh8qqYRNIigUIS2WX71JXBPNSVPDVrDgLzTor1fkI3u?=
 =?us-ascii?Q?C0vxyi/8HqUkTtXcUYpgqTd0Wterp6Nvdm/Eycg3sxQuY/CekTJ8UiiOqQsz?=
 =?us-ascii?Q?WDtnVg04jIrQN+lU0IELeMKFoCcONKfthjtaaGGjtgYOIuuFRXaAW/bTl/ad?=
 =?us-ascii?Q?e9zqABZhEgVQjujYosVgc+H1tjOoPl1WsYziP/Q+kTBRj3aWjjd6fTDf7yU9?=
 =?us-ascii?Q?h7NYaUq+es1lyTf9rrMQ7aXe9FqXXxW/2UScPh09HULo2MqLO4ZBgzBQ6k7x?=
 =?us-ascii?Q?x9kmhGW6cO8lOxg0SS5bLVAzORzr0Ov7dueveVanI1ft+Gw3Azx2EITmT0ER?=
 =?us-ascii?Q?HItRIQNPbAxwWNzWKr7sjmZr5DAHl0CoFaW7KHJgw6HpEKUilO1nbO45M9/u?=
 =?us-ascii?Q?8h1vPSrmhDuOoo13RBMIt4qSoeqk3c1sKcms2VzjzijkPpa5Lmyq1FsSa4EY?=
 =?us-ascii?Q?EoABI+2NEs6fLH4mqZlGclZukwsxWIGWybDKeDQmzXr4qn+39LvoSOPrz+Za?=
 =?us-ascii?Q?ilLBkuidoqQnBajjqNug1rfpDc0hNNMQ4hewYWljpi9cHXLaar+0NV3+aJuh?=
 =?us-ascii?Q?/GJnQoa22BkxyBRwU8UJHs8t9FBSuXHynxrDsjjQTJeAQ0nPkQc8rNe4Y8UK?=
 =?us-ascii?Q?sJhqEryxtdU3JBOtGXwkjW52NTmx4xH8fDVRR06y/OS1Epi+KwdKAuboxcfV?=
 =?us-ascii?Q?urQ+7gSLQl0EOiIj7+PW2VRAho++6KPBJcJ/0XxpGWP3ttABg5lGpc2ib6y7?=
 =?us-ascii?Q?qc3yo5U/I9ZGw2jGlGOIh58JDYuJIfAHKOvMqdtN?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0c48eb49-9330-44a7-4103-08db08bebb07
X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Feb 2023 03:52:27.7525
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: E0NFf2hGkCrNGu0iq0XSZhVOEQKqIvGIvdniQ66H48mQZu4B8SaSb8v+OLESLear9jD3elhFVHHc3CCDkQoxeQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5176
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Liu, Yi L <yi.l.liu@intel.com>
> Sent: Monday, February 6, 2023 5:05 PM
>=20
> Allow the vfio_device file to be in a state where the device FD is
> opened but the device cannot be used by userspace (i.e. its .open_device(=
)
> hasn't been called). This inbetween state is not used when the device
> FD is spawned from the group FD, however when we create the device FD
> directly by opening a cdev it will be opened in the blocked state.
>=20
> The reason for the inbetween state is userspace only gets a FD but

...is "that" userspace...

> doesn't have the secure until binding the FD to an iommufd. So in the

"doesn't gain access permission until..."

> blocked state, only the bind operation is allowed, other device accesses

remove the last part which duplicates with "only ... is allowed"

> are not allowed. Completing bind will allow user to further access the
> device.
>=20
> This is implemented by adding a flag in struct vfio_device_file to mark
> the blocked state and using a simple smp_load_acquire() to obtain the
> flag value and serialize all the device setup with the thread accessing
> this device.
>=20
> Following this lockless scheme, it can safely handle the device FD
> unbound->bound but it cannot handle bound->unbound. To allow this we'd
> need to add a lock on all the vfio ioctls which seems costly. So once
> device FD is bound, it remains bound until the FD is closed.
>=20
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Yi Liu <yi.l.liu@intel.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>