Received-SPF: pass (google.com: domain of linux-crypto+bounces-1937-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: "Zeng, Xin" <xin.zeng@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: "herbert@gondor.apana.org.au" <herbert@gondor.apana.org.au>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	"linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, qat-linux <qat-linux@intel.com>,
	"Cao, Yahui" <yahui.cao@intel.com>
Subject: RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF
 devices
Thread-Topic: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF
 devices
Thread-Index: AQHaVSVFJ9K0TVMQD0C9btTtb+JmirD9TRgAgAQ3vqA=
Date: Fri, 9 Feb 2024 08:23:32 +0000
Message-ID: <DM4PR11MB550222F7A5454DF9DBEE7FEC884B2@DM4PR11MB5502.namprd11.prod.outlook.com>
References: <20240201153337.4033490-1-xin.zeng@intel.com>
 <20240201153337.4033490-11-xin.zeng@intel.com>
 <20240206125500.GC10476@nvidia.com>
In-Reply-To: <20240206125500.GC10476@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?jag/xlm1FoWkSrSNiY6EoR0WVLdZQsg1VmN/tEwtX3TfmMgn4v+ypNrv/tUl?=
 =?us-ascii?Q?SiG0Mje1vaadD9WfBs73GEcgwXqATAGi6EC0ULlIu/p7OsbNl0QlGE44RQ0E?=
 =?us-ascii?Q?T0sJGaFMULUl5KiQYOq+uuojKjDZrUNf4TAbO2x156VQoizpx0B39RtwgSyA?=
 =?us-ascii?Q?jrfQpofxNTO+RSMwu7c9HS5rzTLruxIvjrUWIH3r3i1BP//QpqQ9ZWGMwbcz?=
 =?us-ascii?Q?uSr4zJ4KdeDMCNXH1ta1IahW/rGmpLCKDnIhFskzGB41VVPji3pETMMuHSqP?=
 =?us-ascii?Q?2MS9quA9Wjzcimow06CMVvoBj4iUtUHC0K0RQJbaSmLc4z0k2LXSIpgsPwaN?=
 =?us-ascii?Q?DWRUjBaVMAI3Cex3zKge9Ot46bwO1+spar3JB2d9QuU5jfdcrYnObQA87Kg+?=
 =?us-ascii?Q?Cs0JCRnU//7ZkajWdRHrdIlmGNMFQkOZfupG2dnQ2VTui4kT/YlbDatysoaM?=
 =?us-ascii?Q?AST8FyNl1Kfxy9oEYXjVbALIPeJLexxnEAQ/bDr6szKjuxafBDcQ0ZYWhGKd?=
 =?us-ascii?Q?Ft8pPHUwukWlZXQ7CL78/du1W+v/Fz+eY7H/9cZfWT4pxQzfos/MBauIz/Ck?=
 =?us-ascii?Q?Rxb/YYZXCcMUbx7qiCyCr6kXnau0BSWKhcpdX25koz9A9m1bn5ep8Lw+/vB4?=
 =?us-ascii?Q?c+/yD5DnsTcbZ3b8uiSc+qdAPwGBVFtDun+NMK2vyuCUprYxylaYyAQ08qz2?=
 =?us-ascii?Q?X1GMBHUzVrIEwb3rGic3+ygXUWtXq5WSSwpMXo1Iloug2NPeGIMMEgtOgEK+?=
 =?us-ascii?Q?gV8iLMU/q1ymRRv5vSzaOPPGIw77CLhKJ1zw362tdSIBiIEOgvyozMh0CJ5J?=
 =?us-ascii?Q?zBJ1qYMCgSK7pc2ZTYTcSXpoTM7zV7fxjoWirhgAKX8Yx5moIGSYLfRs3EnE?=
 =?us-ascii?Q?BMLUAucpsdTgbHS9eU3BGW5wpNB25ImrRCKhO4wnK1TzJdNgkjNGMdDojH9N?=
 =?us-ascii?Q?ALWwaZ/9dajezF7SYtcZp7ss9SiQ5vqfB987YgoPU15QbZmreCiwiqwHcnQx?=
 =?us-ascii?Q?aJihytxbQ9AZYtibOjHHQpHQGGsPJmwapkgKlY/aY1AgVGLzh3gqZBLMtXET?=
 =?us-ascii?Q?+iqyzAnfBbBE2OTUMXaxFOvcyK1fBlPye2zI9k5h5ZNULMVFDCyJ6XiejkQT?=
 =?us-ascii?Q?8Js/sy9iqmtyrqJuO9B70AgIiNEuubqgclz56TZpeLo1jTAytfaWyyXbU1Pg?=
 =?us-ascii?Q?T0QGIpt9BO1zVK93ley4coANZP5Slr84bviUPS8PNQFipuDaQ292DLb0PMrD?=
 =?us-ascii?Q?dm4ugRYILTXUVrQjeArgiznUIFN2uH0s8xKMNaSQAp7inzUTmbqdmxVfSN37?=
 =?us-ascii?Q?qfZP9jNvaUi54c/bk29Xj7q9pT/fWi2JdXwRSRuIQjTGe1wSocN/q7nrsFal?=
 =?us-ascii?Q?jlXOCupUVG6oAxBsopjFaoerL9PmB5LevYxl5CXP8fXLTJ8SSL4u1PpMsyBh?=
 =?us-ascii?Q?W3zOcAVr46XgJDtntP7+0FWQ562LCNV1a3N5V7QlcuuG2stFWUWNeSWwupZs?=
 =?us-ascii?Q?n8Jj9V07OGvBWelNKCt/MJK12Wx+7s+1CJS32PnuAJEug3MG2VlZMg1bAxBh?=
 =?us-ascii?Q?2PuZQiulUQXpxkrha3HBj8lCjMZx+oFJqgXSRof6?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-crypto@vger.kernel.org
List-Id: <linux-crypto.vger.kernel.org>
List-Subscribe: <mailto:linux-crypto+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-crypto+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5502.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5a1ecf9e-70e5-46a7-ae21-08dc2948672f
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Feb 2024 08:23:32.4787
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Yw3MXR+2PjQXZeYBX1V+CFAekv/zC7AKwNGt2cfRNPcy17ALOwvdkUTpGKxiB+NcCU0mOKhuU2J1amwJJJyyGg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7499
X-OriginatorOrg: intel.com

Thanks for your comments, Jason.
On Tuesday, February 6, 2024 8:55 PM, Jason Gunthorpe <jgg@nvidia.com> wrot=
e:
> > +
> > +	ops =3D mdev->ops;
> > +	if (!ops || !ops->init || !ops->cleanup ||
> > +	    !ops->open || !ops->close ||
> > +	    !ops->save_state || !ops->load_state ||
> > +	    !ops->suspend || !ops->resume) {
> > +		ret =3D -EIO;
> > +		dev_err(&parent->dev, "Incomplete device migration ops
> structure!");
> > +		goto err_destroy;
> > +	}
>=20
> Why are there ops pointers here? I would expect this should just be
> direct function calls to the PF QAT driver.

I indeed had a version where the direct function calls are Implemented in
QAT driver, while when I look at the functions, most of them=20
only translate the interface to the ops pointer. That's why I put
ops pointers directly into vfio variant driver.

>=20
> > +static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
> > +{
> > +	struct qat_vf_core_device *qat_vdev =3D qat_vf_drvdata(pdev);
> > +
> > +	if (!qat_vdev->core_device.vdev.mig_ops)
> > +		return;
> > +
> > +	/*
> > +	 * As the higher VFIO layers are holding locks across reset and using
> > +	 * those same locks with the mm_lock we need to prevent ABBA
> deadlock
> > +	 * with the state_mutex and mm_lock.
> > +	 * In case the state_mutex was taken already we defer the cleanup wor=
k
> > +	 * to the unlock flow of the other running context.
> > +	 */
> > +	spin_lock(&qat_vdev->reset_lock);
> > +	qat_vdev->deferred_reset =3D true;
> > +	if (!mutex_trylock(&qat_vdev->state_mutex)) {
> > +		spin_unlock(&qat_vdev->reset_lock);
> > +		return;
> > +	}
> > +	spin_unlock(&qat_vdev->reset_lock);
> > +	qat_vf_state_mutex_unlock(qat_vdev);
> > +}
>=20
> Do you really need this? I thought this ugly thing was going to be a
> uniquely mlx5 thing..

I think that's still required to make the migration state synchronized
if the VF is reset by other VFIO emulation paths. Is it the case?=20
BTW, this implementation is not only in mlx5 driver, but also in other
Vfio pci variant drivers such as hisilicon acc driver and pds driver.
Thanks,
Xin