Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Liu, Yi L" <yi.l.liu@intel.com>
To:     "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>
CC:     "jgg@nvidia.com" <jgg@nvidia.com>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "farman@linux.ibm.com" <farman@linux.ibm.com>,
        "pmorel@linux.ibm.com" <pmorel@linux.ibm.com>,
        "borntraeger@linux.ibm.com" <borntraeger@linux.ibm.com>,
        "frankja@linux.ibm.com" <frankja@linux.ibm.com>,
        "imbrenda@linux.ibm.com" <imbrenda@linux.ibm.com>,
        "david@redhat.com" <david@redhat.com>,
        "akrowiak@linux.ibm.com" <akrowiak@linux.ibm.com>,
        "jjherne@linux.ibm.com" <jjherne@linux.ibm.com>,
        "pasic@linux.ibm.com" <pasic@linux.ibm.com>,
        "zhenyuw@linux.intel.com" <zhenyuw@linux.intel.com>,
        "Wang, Zhi A" <zhi.a.wang@intel.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "intel-gvt-dev@lists.freedesktop.org" 
        <intel-gvt-dev@lists.freedesktop.org>,
        "intel-gfx@lists.freedesktop.org" <intel-gfx@lists.freedesktop.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] kvm/vfio: Fix potential deadlock on vfio group_lock
Thread-Topic: [PATCH] kvm/vfio: Fix potential deadlock on vfio group_lock
Thread-Index: AQHZLOCowN6l2rEgl0uw/AQRWmSbkK6naAtg
Date:   Fri, 20 Jan 2023 15:08:58 +0000
Message-ID: <DS0PR11MB7529B08476DF1764F1B6C007C3C59@DS0PR11MB7529.namprd11.prod.outlook.com>
References: <20230120150528.471752-1-yi.l.liu@intel.com>
In-Reply-To: <20230120150528.471752-1-yi.l.liu@intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?MazrZjcZ8uUDc+iQNHxAu882f+kbfnXMSuHsvfR5NFBxEIwUS/amMuJEwoMR?=
 =?us-ascii?Q?XPnkRSVn2npV6XmH/YeGRAmGkUBAhaq0dkgVKGoBujY6HZixEd1iGolBhrQA?=
 =?us-ascii?Q?sXzUkoyrIV0cHhu3WT+YjDG7eb89xDo7RHtVvP0JV6CQIKkXBCn0av/Gh5/U?=
 =?us-ascii?Q?8xY9QO0ZlD9Qnu4uXpVKhnwxc7Tapl6UailEPUJ14V6UwCEpWyMQHgRhGU7Q?=
 =?us-ascii?Q?ZxV0HMy9lNuWO3x5ZOqxC5PoabuBK7WNcs1O1vbhltZOEPnR1HOijm5o/X7v?=
 =?us-ascii?Q?wey/h9490ypYQHgWvmTDnsjqHSCnzDd4lsMPr3qvEyAEOY2Jxj0g7mHziwgd?=
 =?us-ascii?Q?aCKeCW8q+ZpvxZcqUtg5cMZw+0dS6IOw8f32pcQWkx6EOOFqHljeWyrHvKbw?=
 =?us-ascii?Q?8SgYbRDm8fHnZeLJWZREJj02otDr7UwTvZvHFC0zaGFpJxQgtiwWS7lWORGc?=
 =?us-ascii?Q?Z+H8FV3I50MCEZzV4yx2vDtWTHBWXh5NV+Pec2j0MK4LcdHo6bAJ4lh3evPm?=
 =?us-ascii?Q?vuO52guYx+6DZaGlwOIPJdkGPg4IhVOGbNg6+63jsdti0KIk0nKR4m0b0WCI?=
 =?us-ascii?Q?iepEJgz1b2akdeh2XGTMEeh3L3TzGwr1Ff3SPpByyKVBirzqPJSPgRru2T3H?=
 =?us-ascii?Q?tddtkWdecv/EElegmrgXwS2vepm6/V5tS/7tBOTp6wO4zp6GErMnizz0ZUTy?=
 =?us-ascii?Q?WB+XKaaTjcsFigOMcskK9X6DC9mFuwJZ9wGB16/lk73PQVOnADshW/K1fbSI?=
 =?us-ascii?Q?TomAczcNgyuNDYG4bLyeNN8ae1J8cVqBwiG7d5CGoVuTMoohkGr8S3TiSzXm?=
 =?us-ascii?Q?YKqRVC/mtR8BuKzXBLDKWQEP4af2P0uDdnK09mAzPFrVrNFJ6AQ1VKnyz/OP?=
 =?us-ascii?Q?sPoMPDbJk51GyjkeYDPxQaAhd42NFG/BWPeXNdo6MaeqeTf0nQ/4vpd2WU3Z?=
 =?us-ascii?Q?tGYF7sFNXG4+PMTX937i+2l1hInuAkwMoipbMAcDJ8y0QDVASUhNR6edm45J?=
 =?us-ascii?Q?rsoxSB6RBWa5rx1/mN7Iesy64WsqMnMiDwJI30pJhweP2knoxJ2IpmSm6hR1?=
 =?us-ascii?Q?5gvWvtznaHPhIIvnq4uZR6WvchcQ6sb+WyoyB4HqZjMPXgttZH33Z9TWshw7?=
 =?us-ascii?Q?SOQ3SvQ6VT5pX3tl7woOLMTBuD7E9WQ+MD+ThvjO4hyQZMdujxnIg544j/4p?=
 =?us-ascii?Q?qKSYV/t/CRW3E5faiZJrSH7xnPlRzZXkxY57wVmblEPUwju4Dqf51rC/xj3p?=
 =?us-ascii?Q?xkaPE1aETyKUXvhO+yXPsXgsYXJyfFitRy25SvI8GceDpFq08jH+6DoJgKX8?=
 =?us-ascii?Q?f/coFUa6pyXHBaU36mCUzkiHCAeLw0NRi1va3W0TKlVB2YO3sdS8ffe7suoH?=
 =?us-ascii?Q?w2tNH0Yr9CDRN6AHpSJnU4t4phSUB55KZBGoW2bHYn8oDszsdJlyUR6AMmzK?=
 =?us-ascii?Q?LJ5/KcUrjKT5H+PmOsLjJb3kyJIQh0Ld/YzTEfVxNFW92mXqAbot40ry3eDR?=
 =?us-ascii?Q?Eh9QSs320m27FWPGlY52W5VTq6Q9ffIT7/muvC3tZDVAxQvJHd4VB1cOOmwv?=
 =?us-ascii?Q?uu5O4mQScwrX+/s42kgxlKb18bi2qodDWh5qiUUQ?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78f7b398-8f01-460b-c422-08dafaf841c4
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jan 2023 15:08:58.7968
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: lKDvuco3RjJb8nE/n1I7UkMrTS0iRA08YhrRCJz5anABsH+5b0ArCLyAAmkGvOz97AFi+L21iWurdBv1drLkFA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5832
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Liu, Yi L <yi.l.liu@intel.com>
> Sent: Friday, January 20, 2023 11:05 PM
>=20
> Currently it is possible that the final put of a KVM reference comes from
> vfio during its device close operation.  This occurs while the vfio group
> lock is held; however, if the vfio device is still in the kvm device list=
,
> then the following call chain could result in a deadlock:
>=20
> VFIO holds group->group_lock/group_rwsem
>   -> kvm_put_kvm
>    -> kvm_destroy_vm
>     -> kvm_destroy_devices
>      -> kvm_vfio_destroy
>       -> kvm_vfio_file_set_kvm
>        -> vfio_file_set_kvm
>         -> try to hold group->group_lock/group_rwsem
>=20
> The key function is the kvm_destroy_devices() which triggers destroy cb
> of kvm_device_ops. It calls back to vfio and try to hold group_lock. So
> if this path doesn't call back to vfio, this dead lock would be fixed.
> Actually, there is a way for it. KVM provides another point to free the
> kvm-vfio device which is the point when the device file descriptor is
> closed. This can be achieved by providing the release cb instead of the
> destroy cb. Also rename kvm_vfio_destroy() to be kvm_vfio_release().
>=20
> 	/*
> 	 * Destroy is responsible for freeing dev.
> 	 *
> 	 * Destroy may be called before or after destructors are called
> 	 * on emulated I/O regions, depending on whether a reference is
> 	 * held by a vcpu or other kvm component that gets destroyed
> 	 * after the emulated I/O.
> 	 */
> 	void (*destroy)(struct kvm_device *dev);
>=20
> 	/*
> 	 * Release is an alternative method to free the device. It is
> 	 * called when the device file descriptor is closed. Once
> 	 * release is called, the destroy method will not be called
> 	 * anymore as the device is removed from the device list of
> 	 * the VM. kvm->lock is held.
> 	 */
> 	void (*release)(struct kvm_device *dev);
>=20
> Fixes: 421cfe6596f6 ("vfio: remove VFIO_GROUP_NOTIFY_SET_KVM")
> Reported-by: Alex Williamson <alex.williamson@redhat.com>
> Suggested-by: Kevin Tian <kevin.tian@intel.com>
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Yi Liu <yi.l.liu@intel.com>

More background can be found in Mathew's work.
https://lore.kernel.org/kvm/20230114000351.115444-1-mjrosato@linux.ibm.com/=
T/#u

Regards,
Yi Liu