From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Matthew Rosato <mjrosato@linux.ibm.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "jgg@nvidia.com" <jgg@nvidia.com>
CC:     "cohuck@redhat.com" <cohuck@redhat.com>,
        "farman@linux.ibm.com" <farman@linux.ibm.com>,
        "pmorel@linux.ibm.com" <pmorel@linux.ibm.com>,
        "borntraeger@linux.ibm.com" <borntraeger@linux.ibm.com>,
        "frankja@linux.ibm.com" <frankja@linux.ibm.com>,
        "imbrenda@linux.ibm.com" <imbrenda@linux.ibm.com>,
        "david@redhat.com" <david@redhat.com>,
        "akrowiak@linux.ibm.com" <akrowiak@linux.ibm.com>,
        "jjherne@linux.ibm.com" <jjherne@linux.ibm.com>,
        "pasic@linux.ibm.com" <pasic@linux.ibm.com>,
        "zhenyuw@linux.intel.com" <zhenyuw@linux.intel.com>,
        "Wang, Zhi A" <zhi.a.wang@intel.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "intel-gvt-dev@lists.freedesktop.org" 
        <intel-gvt-dev@lists.freedesktop.org>,
        "intel-gfx@lists.freedesktop.org" <intel-gfx@lists.freedesktop.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v4 1/2] vfio: fix deadlock between group lock and kvm lock
Thread-Topic: [PATCH v4 1/2] vfio: fix deadlock between group lock and kvm
 lock
Thread-Index: AQHZOBmQ43ni2TX4OkKomzgfxOFFkq7BTI4w
Date:   Mon, 6 Feb 2023 03:52:25 +0000
Message-ID: <BN9PR11MB5276544E7CE1289C6A6FF4098CDA9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230203215027.151988-1-mjrosato@linux.ibm.com>
 <20230203215027.151988-2-mjrosato@linux.ibm.com>
In-Reply-To: <20230203215027.151988-2-mjrosato@linux.ibm.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?qzOXEKbR3h0De9cFB1Ojqo6ukMV/tBpxWVlMyJV+mEcXJa6mLQ0bYV8AXPe6?=
 =?us-ascii?Q?q/bh1GIUPx3lwTZLEYvlkMxpZ1VY2f33BaQfEoNMG6h17oNVSikOnopAsZf9?=
 =?us-ascii?Q?lUi0MoypeP4w2iuwtP7XrV6nh9DqaNNF97ZaWFFEq/+B0Fntyw6SpmWsDPFC?=
 =?us-ascii?Q?MYNP5TCj/z5rwUJgfIJnoBLpvDHOz9xKFk9y7OTpBHCrMl+/uhVFkiK4fvs8?=
 =?us-ascii?Q?1Y7ZISQ8vur4zbIV7yskgC0UnhCgumwNhWJNE+bMYYlsnYpTMmsWOYtRGfg2?=
 =?us-ascii?Q?2WacG2m4qXZsqNtGOI+IXRkejxy9MescAgM87FonyzGBi/g7WBhaSHlmqdQX?=
 =?us-ascii?Q?wnYJmOMXXIpp0dxAPsxQoRtOlc0jBgvbX/xd/DDOUPvf09XFKRLQ4RT4REan?=
 =?us-ascii?Q?jp/m6TFJcpeNShC7gh/QaSzsXW5DuD1u8ilrL711hgE5WGP+e6w+1Fpb+BKc?=
 =?us-ascii?Q?ZccN8v11UhsrqxPKmxS+Yppw0q9B8gUI2UClAIRPRZe7fNEMISN1rX+lkeX4?=
 =?us-ascii?Q?TTRXQwJiI3KmAOFWKhv4MhkNNdrZOLCNGQsrERfr0wCQ2csXaQRYPSsoZiaS?=
 =?us-ascii?Q?FYSS5kxzsD3YFVBby8Y+hjFci1bj1uwRh5dXw+KPnkYyXa1FrA4GXSMjCNIG?=
 =?us-ascii?Q?YXE9mWOPGcSzANxUQPCeRMF7QhMpgIYQ0uflD0m5duXyqbfSnu+vem+UVfIX?=
 =?us-ascii?Q?Icm48lgVLHOu4Q+foWyrL7g8uEFUrrK4YLTYH1BYNzL5JMOEcKtMbS8g/Skk?=
 =?us-ascii?Q?24Q9ksNTMNy+kUkJQojrpkcP8bb1YFbewwgpmvlm9L0znN4IbVMrM8ZGfreh?=
 =?us-ascii?Q?U0PoeHoC4CE3UGI5m1Gunysfg0/YWzTRg8KpsGpOLhyzMHsTzJBPADUiWyX6?=
 =?us-ascii?Q?J2YBQNgBeb5gPyXjv4EVUedHNlYoY8X5k7uxaayf9DUXYStQW1jhAgv0TrnG?=
 =?us-ascii?Q?eEzYnyVWr2MZfE0/8UYkGOa2SCHm6vrwKww4t4Js99LSRpuqIpIyqy0rIck9?=
 =?us-ascii?Q?zPuYXoz22068UeQkGuwtJHEz2DnCxTDUFF2K2PC1OMAFJQcQz6t1RuwjlJZq?=
 =?us-ascii?Q?d6g/FOZLVqaLq0hIFstKwtjSIOQnpVfmlbly/+lnowYblBSLOPtjeOqK4dZD?=
 =?us-ascii?Q?cyqTQO8zboKk26IAaTrnGwXDCNZAPwhNZwvZ5Z5QTLDwTQFEYsaHOdq40Gck?=
 =?us-ascii?Q?rFBQiWMBZECu/e9H0dlVmfgxwp9N/y0hKhamhzNRflXk3DLOR8lMVZG1svm+?=
 =?us-ascii?Q?MNoHqeDeHLZrNi8XSCYJU0vF9FAZtcoMjbWMh5fZDp13HVkMaCoLJfL2iAUc?=
 =?us-ascii?Q?9Y98PjKSFVqXfvUl2w2O6bJ5/mUfvoyeIfCz4Yy/43TjhpIHMfKfurDZEDZO?=
 =?us-ascii?Q?8w8NtSr2YYylDg0N3HYFT1v6YhwtLuQ+WKP9nm4QyAMFikrHmUy6QWK7oFde?=
 =?us-ascii?Q?A74bhEtxfif0cUSIsLrcAI9xEGqATZMfgzkCBE52Ji3BpLKLPDevyb7V6nO8?=
 =?us-ascii?Q?BHby6mZCwX5QOcnfdegT6bNBn6vYFuZ6X5PrSz5UNqbm0cQgSj2g9PzPAdNU?=
 =?us-ascii?Q?mdpoNCBBzKgX0rM7RCOS5ZfXlac+3DPG2jLHMhVr?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a3efa97d-95bd-48c9-957f-08db07f58f8d
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Feb 2023 03:52:25.9704
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: O2PpKrX7m2gVyJjUxUaGPMJAPKy9sRHKf/e6vfrS4MwPf05NkD++dx85DgYK5IR2jXfL9y0Hi8cCZM8oNBFpOg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7810
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Matthew Rosato <mjrosato@linux.ibm.com>
> Sent: Saturday, February 4, 2023 5:50 AM
>=20
> After 51cdc8bc120e, we have another deadlock scenario between the
> kvm->lock and the vfio group_lock with two different codepaths acquiring
> the locks in different order.  Specifically in vfio_open_device, vfio
> holds the vfio group_lock when issuing device->ops->open_device but some
> drivers (like vfio-ap) need to acquire kvm->lock during their open_device
> routine;  Meanwhile, kvm_vfio_release will acquire the kvm->lock first
> before calling vfio_file_set_kvm which will acquire the vfio group_lock.
>=20
> To resolve this, let's remove the need for the vfio group_lock from the
> kvm_vfio_release codepath.  This is done by introducing a new spinlock to
> protect modifications to the vfio group kvm pointer, and acquiring a kvm
> ref from within vfio while holding this spinlock, with the reference held
> until the last close for the device in question.
>=20
> Fixes: 51cdc8bc120e ("kvm/vfio: Fix potential deadlock on vfio group_lock=
")
> Reported-by: Anthony Krowiak <akrowiak@linux.ibm.com>
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>