Received-SPF: pass (google.com: domain of linux-crypto+bounces-1939-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Fri, 9 Feb 2024 08:10:45 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Zeng, Xin" <xin.zeng@intel.com>
Cc: "herbert@gondor.apana.org.au" <herbert@gondor.apana.org.au>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"Tian, Kevin" <kevin.tian@intel.com>,
	"linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	qat-linux <qat-linux@intel.com>, "Cao, Yahui" <yahui.cao@intel.com>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF
 devices
Message-ID: <20240209121045.GP10476@nvidia.com>
References: <20240201153337.4033490-1-xin.zeng@intel.com>
 <20240201153337.4033490-11-xin.zeng@intel.com>
 <20240206125500.GC10476@nvidia.com>
 <DM4PR11MB550222F7A5454DF9DBEE7FEC884B2@DM4PR11MB5502.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <DM4PR11MB550222F7A5454DF9DBEE7FEC884B2@DM4PR11MB5502.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?pVzv91DqIGTpjySVeuBSireeQEL8OxOfRNhGTYV+iQYU9RoyGTap8OKFRokO?=
 =?us-ascii?Q?Q9Oc+6zDIJ4+CQzjt0nuk48QPQe4encgcIhPaxmz5Oplhs05oLCgtXeoD8pK?=
 =?us-ascii?Q?C68L+/fYS0xBukgELqaPGNYKPrKMnXPesmLKTyeOs71ke737aX4BHi6+NS2H?=
 =?us-ascii?Q?lv/VDkR8efV8ZaYuF2O/JVkPvyo/utGyrL1oFz6xeXutEpCnuvP9qrQ6Q2OE?=
 =?us-ascii?Q?O/3+zdboOCn/zzJPf5Z546Ik5ejZZxomJ19A4yPfIkqDvp7iqCDH0fYSN/vo?=
 =?us-ascii?Q?czkJ8PPE86JFu+ZePJ1xEh5xJHhf/R7hEsVRT4PDwgETyA5NulCkn6ibj0jx?=
 =?us-ascii?Q?2T0gOev58jQotjWnGRPw8LCN5HqAB9hOITtbuQ3wtd0GHIkU5t7VvVznpxjL?=
 =?us-ascii?Q?ewvXuOOmRhZBq05ILeBsRsYwdInRT2jG5NFU314NUb3UaWXoXhHYTIKxsj5a?=
 =?us-ascii?Q?iQK8q9/MR2ERpUt6EhWYRG9nlithVkuedPP+9UKAP3g3+N5eDG7Z+y6Kd5sh?=
 =?us-ascii?Q?7k3Dsu2JkTVdAAmDpPo/j2+wvj/o9LzuOuhA6wCqKAO48Eu2pHKlXfEaJbEM?=
 =?us-ascii?Q?S0lX9u4fAvH7PUIEB7ZPINtFQ9GOCrIt0zySX5X+BVQxP2DFRnxGBPRSUyY8?=
 =?us-ascii?Q?PKiqQmcBPjb0RLAdjbAoDBBiumncNrf7A7Qbwa2wntbeMLU++rd39wfq3c5I?=
 =?us-ascii?Q?XQjl2sA53b7YbwCa776f3Jnv3zckcJuJIpA4RNB4EdTIVsO4NrwM0s1GLjEu?=
 =?us-ascii?Q?lwByaxUfjjtzlADXkJCrQ+BQkWPEII/Ng14Ghmuh7spwaqjUYQThOo4fba4K?=
 =?us-ascii?Q?lPonrhccgGbpYK5PegrzSmoVNUazyxMG5SLp4u5WxqxzQtgpax+/+gHoOrxD?=
 =?us-ascii?Q?A12e4jDdnfYpxH4oO2zppzKaM+kCUlyo3KjfuhK27hjGuc9nTVqOhUOlaGdv?=
 =?us-ascii?Q?cjJur3EAY5bUt+H9fLTIx9tnm8/7BKAYx/kDCH6zlrS7x7l+g3C1g5c6EVLJ?=
 =?us-ascii?Q?wc9rE3N9edU3JcdHsN72/S/ADpeG2Y/bS/eNjBxBhxQdtIQjJpz4wzzg+QsX?=
 =?us-ascii?Q?XRvzWkmEMuj9PCmfkBW4bXtzv0YX6HqjP0UZRLRf8WSxteaQl4kXLYl6zkvg?=
 =?us-ascii?Q?ZLSXQ8N036Plw+sp+QhNUKQU0X5AJV3+U2wWTJjf1j/dWYAOXpdz3tDsS/tv?=
 =?us-ascii?Q?ka8Y7KUtaFd+PX01kSvMrxCLBownfjnzW3QnD56tMRLNZD02B8jbY1WXLlu3?=
 =?us-ascii?Q?x02CEiz5us/4A6VvorvXloOqxan5wxhXnJAivDsWEgaZ8jYcaAqAWiSXpxaI?=
 =?us-ascii?Q?NMar2XBxHZBYqjaVOSNFkEfa7V5ke6aPbjt1a7dCHNpbUivuREtYTHl5gAjN?=
 =?us-ascii?Q?atYmig2cmB5+x5a80cVjN3xqQnCO7vx5j1qHQX51/ezPSnKwnvDk5jStoa33?=
 =?us-ascii?Q?1uqj7P8ZATRcOvObR4uHagvzD4GvyZ8v1Qg731DLzwHq0CWUJ/XC5X6tbBD0?=
 =?us-ascii?Q?/hkLePz31BQP0xNWdIh3w40ZB0Cz1yd2BkoUyseJ1sQEFgXfVo3qot8VBL9D?=
 =?us-ascii?Q?vyTpM8OwnICsI747ywsQvHoiDiySHCoCBddQXZgs?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 31e3291b-6b5d-451b-cd3f-08dc296825cf
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 12:10:47.0697
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mKnkYFO6aD7cnXDw6jAttqkyZnwJGdJfcc8rhv1O5Ujm4gNVV8ntR48OFqyHL1qw
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5304

On Fri, Feb 09, 2024 at 08:23:32AM +0000, Zeng, Xin wrote:
> Thanks for your comments, Jason.
> On Tuesday, February 6, 2024 8:55 PM, Jason Gunthorpe <jgg@nvidia.com> wrote:
> > > +
> > > +	ops = mdev->ops;
> > > +	if (!ops || !ops->init || !ops->cleanup ||
> > > +	    !ops->open || !ops->close ||
> > > +	    !ops->save_state || !ops->load_state ||
> > > +	    !ops->suspend || !ops->resume) {
> > > +		ret = -EIO;
> > > +		dev_err(&parent->dev, "Incomplete device migration ops
> > structure!");
> > > +		goto err_destroy;
> > > +	}
> > 
> > Why are there ops pointers here? I would expect this should just be
> > direct function calls to the PF QAT driver.
> 
> I indeed had a version where the direct function calls are Implemented in
> QAT driver, while when I look at the functions, most of them 
> only translate the interface to the ops pointer. That's why I put
> ops pointers directly into vfio variant driver.

But why is there an ops indirection at all? Are there more than one
ops?

> > > +static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
> > > +{
> > > +	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
> > > +
> > > +	if (!qat_vdev->core_device.vdev.mig_ops)
> > > +		return;
> > > +
> > > +	/*
> > > +	 * As the higher VFIO layers are holding locks across reset and using
> > > +	 * those same locks with the mm_lock we need to prevent ABBA
> > deadlock
> > > +	 * with the state_mutex and mm_lock.
> > > +	 * In case the state_mutex was taken already we defer the cleanup work
> > > +	 * to the unlock flow of the other running context.
> > > +	 */
> > > +	spin_lock(&qat_vdev->reset_lock);
> > > +	qat_vdev->deferred_reset = true;
> > > +	if (!mutex_trylock(&qat_vdev->state_mutex)) {
> > > +		spin_unlock(&qat_vdev->reset_lock);
> > > +		return;
> > > +	}
> > > +	spin_unlock(&qat_vdev->reset_lock);
> > > +	qat_vf_state_mutex_unlock(qat_vdev);
> > > +}
> > 
> > Do you really need this? I thought this ugly thing was going to be a
> > uniquely mlx5 thing..
> 
> I think that's still required to make the migration state synchronized
> if the VF is reset by other VFIO emulation paths. Is it the case? 
> BTW, this implementation is not only in mlx5 driver, but also in other
> Vfio pci variant drivers such as hisilicon acc driver and pds
> driver.

It had to specifically do with the mm lock interaction that, I
thought, was going to be unique to the mlx driver. Otherwise you could
just directly hold the state_mutex here.

Yishai do you remember the exact trace for the mmlock entanglement?

Jason