Received-SPF: pass (google.com: domain of linux-crypto+bounces-2033-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: "Zeng, Xin" <xin.zeng@intel.com>
To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
CC: "linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, qat-linux <qat-linux@intel.com>,
	"herbert@gondor.apana.org.au" <herbert@gondor.apana.org.au>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>, "jgg@nvidia.com"
	<jgg@nvidia.com>, "yishaih@nvidia.com" <yishaih@nvidia.com>, "Tian, Kevin"
	<kevin.tian@intel.com>, "Cao, Yahui" <yahui.cao@intel.com>
Subject: RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF
 devices
Thread-Topic: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF
 devices
Thread-Index: AQHaVSVFJ9K0TVMQD0C9btTtb+JmirEAZyqAgAFFQ1A=
Date: Tue, 13 Feb 2024 13:08:47 +0000
Message-ID: <DM4PR11MB55025C3ECE1896D9C5CA4E86884F2@DM4PR11MB5502.namprd11.prod.outlook.com>
References: <20240201153337.4033490-1-xin.zeng@intel.com>
 <20240201153337.4033490-11-xin.zeng@intel.com>
 <972cc8a41a8549d19ed897ee7335f9e0@huawei.com>
In-Reply-To: <972cc8a41a8549d19ed897ee7335f9e0@huawei.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?xUAhQAk3eOXjN1SaZ0vssa/AYqbYyQ+7gM+Na4i/pw70VJR5CgSk4+qJmA0B?=
 =?us-ascii?Q?wEDOT6dAesKj0czvGPCp31N4kNugwH0zTIMQUx8UCQGYRO5Vkmb03PDHixQi?=
 =?us-ascii?Q?ujY6gWdkjXm135e/5gU1Q6ukDps3Pf5fz6l3oaoHeA6PDSKuaCqinvsyGIbL?=
 =?us-ascii?Q?zhDCXbwJXPW66jVUMwQZtMIDMo/VxTmidKMnkVTTyxblwq1SyLDWRFWb2OLu?=
 =?us-ascii?Q?Lx9YGHyAbXCehG4b8JgzLWkVu8S5md+jkuAD/1T0PaSyAn/EcshgXdibm1R1?=
 =?us-ascii?Q?kB0+JMy8dPN0DiUfr8V9RF8Fp602dM4uAd+ghJ76KtLPwBG7iriL4BOqXO39?=
 =?us-ascii?Q?YxEpI4UMMMjPIQM9oct5TWWdb2K8giygz3d2MIfoDWHU8ZXrPc0VCNzBgaRb?=
 =?us-ascii?Q?sVNZzckA/lqAaMmqsKTR6HPW1R/e8rRrQ0B/MCn0Ord3lOFFi9NASh/NdAQl?=
 =?us-ascii?Q?OgeXYvr677IFL+g4Y6L63rk/shKvGMcolfdnhqb0f4IvW7Ht8Dxr75rubBs9?=
 =?us-ascii?Q?UYej2pTYJjDXmZF3IXpuMPQYQO8X8hldy98dUO3frFFdy2GilzMLAIRxy7RN?=
 =?us-ascii?Q?afgeQvmyU9jLLGvUOr5mbaG2LTN2/Unjdq6o+s1zob0V5/WQ+ZtZj2FHkn+u?=
 =?us-ascii?Q?5S9sT01RKjyqDidmnBzMittuhJ/72S4OC2xybm9N/QaQ0qbTrU+0kT9iurFd?=
 =?us-ascii?Q?ZsAGUgcFscWjkFiFDsNwzaEHLJQssAiF5hEDa3sqWvPwt9Dn5safMdysU1k/?=
 =?us-ascii?Q?9YBHW5H+GwbO9BU/urTQWOa5LVJ0VQYCHv+tMmN0VtLU6sAxmO6bYOpxeD05?=
 =?us-ascii?Q?YwgI/Tz1X6kgPT6Y3YgJiLNtulNqnD5q8MRuh9CUAq+mComjgaDUIo58Axir?=
 =?us-ascii?Q?EnUmyB6zeTLL2mkfDHPsNZfFkG/F9SuOswxTloBWfPumbqO+j+hP51TpSIMe?=
 =?us-ascii?Q?Hayk1mrfpI5aa/xc4nj+/OrKZkVsCDXOdUUzVVjXD7rApIae3OmnRYNJMgZY?=
 =?us-ascii?Q?wjtRlxMS425RvO5xNDzueHTwdU3oSwN1zLA0bT1iUwVm60zMmFtqo3sjvyNv?=
 =?us-ascii?Q?W9ijirAi2gXd4dSNmGrJRN65p3t+E1kInOCH+z++bxIEgqEQWqmGNJd6vsdE?=
 =?us-ascii?Q?5WdzEY/kGs+vbq7g/1nm0jT5DkpXricRcUfCclsvU7sgSWRmiw9HySh5+W6P?=
 =?us-ascii?Q?D7sFDX3QPzornlXK0cV6qg1VdWDx12S3/IK6th90VMthPWrDbF7GfpOZ088m?=
 =?us-ascii?Q?7wJqXW/5hDBsElWy7yBnHwf6i4MSHCoG6crRsyV0p6UUC5jXaBEzMMwO8RGA?=
 =?us-ascii?Q?L8v5Vb80VHupb3gPBwH9+sEjbbSSQMCdx1eX50JZeU0DJgnKti7P4jaND7dE?=
 =?us-ascii?Q?VdN0Og8W3oV390V6XKYZRWxHMlfoLABgBrsC/V9kdlQ6j3sMzTHoKbv9i1Zd?=
 =?us-ascii?Q?DrFbWY4EP1VG7QB1Pf9wphvxxwnmEQLOtNlB2SGPp2+wsaNQV7PxiGsNp2RL?=
 =?us-ascii?Q?SwWTikmDYhFVAs5XALCWkVunZshYJzTQ1LpP9h+CUEDxeE4/lSbwQ/5NXbLM?=
 =?us-ascii?Q?mf+aqCvLzD4j6/mYv9ZobNZ1gjE8tndukiaioCe1?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-crypto@vger.kernel.org
List-Id: <linux-crypto.vger.kernel.org>
List-Subscribe: <mailto:linux-crypto+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-crypto+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5502.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ee2c3493-975a-424e-3ed6-08dc2c94ea1a
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Feb 2024 13:08:47.3861
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: q9BVodFQX8pwMSShZVLSKhEbxpq4VA+HxowDqNDTevq9ys1JhfKPsWv3MVRZo0JwW8+MIC5KuQwjJ3ILTBBhvw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4861
X-OriginatorOrg: intel.com

Thanks for the comments, Shameer.

> -----Original Message-----
> From: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Sent: Thursday, February 8, 2024 8:17 PM
> To: Zeng, Xin <xin.zeng@intel.com>; herbert@gondor.apana.org.au;
> alex.williamson@redhat.com; jgg@nvidia.com; yishaih@nvidia.com; Tian, Kev=
in
> <kevin.tian@intel.com>
> Cc: linux-crypto@vger.kernel.org; kvm@vger.kernel.org; qat-linux <qat-
> linux@intel.com>; Cao, Yahui <yahui.cao@intel.com>
> Subject: RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF=
 devices
>=20
>=20
> > +static struct qat_vf_migration_file *
> > +qat_vf_save_device_data(struct qat_vf_core_device *qat_vdev)
> > +{
> > +	struct qat_vf_migration_file *migf;
> > +	int ret;
> > +
> > +	migf =3D kzalloc(sizeof(*migf), GFP_KERNEL);
> > +	if (!migf)
> > +		return ERR_PTR(-ENOMEM);
> > +
> > +	migf->filp =3D anon_inode_getfile("qat_vf_mig", &qat_vf_save_fops,
> > migf, O_RDONLY);
> > +	ret =3D PTR_ERR_OR_ZERO(migf->filp);
> > +	if (ret) {
> > +		kfree(migf);
> > +		return ERR_PTR(ret);
> > +	}
> > +
> > +	stream_open(migf->filp->f_inode, migf->filp);
> > +	mutex_init(&migf->lock);
> > +
> > +	ret =3D qat_vdev->mdev->ops->save_state(qat_vdev->mdev);
> > +	if (ret) {
> > +		fput(migf->filp);
> > +		kfree(migf);
>=20
> Probably don't need that kfree(migf) here as fput() -->  qat_vf_release_f=
ile () will
> do that.

Thanks, it's redundant, will update it in next version.

>=20
> > +static int qat_vf_pci_get_data_size(struct vfio_device *vdev,
> > +				    unsigned long *stop_copy_length)
> > +{
> > +	struct qat_vf_core_device *qat_vdev =3D container_of(vdev,
> > +			struct qat_vf_core_device, core_device.vdev);
> > +
> > +	*stop_copy_length =3D qat_vdev->mdev->state_size;
>=20
> Do we need a lock here or this is not changing?

Yes, will update it in next version.

>=20
> > +	return 0;
> > +}
> > +
> > +static const struct vfio_migration_ops qat_vf_pci_mig_ops =3D {
> > +	.migration_set_state =3D qat_vf_pci_set_device_state,
> > +	.migration_get_state =3D qat_vf_pci_get_device_state,
> > +	.migration_get_data_size =3D qat_vf_pci_get_data_size,
> > +};
> > +
> > +static void qat_vf_pci_release_dev(struct vfio_device *core_vdev)
> > +{
> > +	struct qat_vf_core_device *qat_vdev =3D container_of(core_vdev,
> > +			struct qat_vf_core_device, core_device.vdev);
> > +
> > +	qat_vdev->mdev->ops->cleanup(qat_vdev->mdev);
> > +	qat_vfmig_destroy(qat_vdev->mdev);
> > +	mutex_destroy(&qat_vdev->state_mutex);
> > +	vfio_pci_core_release_dev(core_vdev);
> > +}
> > +
> > +static int qat_vf_pci_init_dev(struct vfio_device *core_vdev)
> > +{
> > +	struct qat_vf_core_device *qat_vdev =3D container_of(core_vdev,
> > +			struct qat_vf_core_device, core_device.vdev);
> > +	struct qat_migdev_ops *ops;
> > +	struct qat_mig_dev *mdev;
> > +	struct pci_dev *parent;
> > +	int ret, vf_id;
> > +
> > +	core_vdev->migration_flags =3D VFIO_MIGRATION_STOP_COPY |
> > VFIO_MIGRATION_P2P;
> > +	core_vdev->mig_ops =3D &qat_vf_pci_mig_ops;
> > +
> > +	ret =3D vfio_pci_core_init_dev(core_vdev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	mutex_init(&qat_vdev->state_mutex);
> > +	spin_lock_init(&qat_vdev->reset_lock);
> > +
> > +	parent =3D qat_vdev->core_device.pdev->physfn;
>=20
> Can we use pci_physfn() here?

Sure, will update it in next version.

>=20
> > +	vf_id =3D pci_iov_vf_id(qat_vdev->core_device.pdev);
> > +	if (!parent || vf_id < 0) {
>=20
> Also if the pci_iov_vf_id() return success I don't think you need to
> check for parent and can use directly below.

OK, will update it in next version.

>=20
> > +		ret =3D -ENODEV;
> > +		goto err_rel;
> > +	}
> > +
> > +	mdev =3D qat_vfmig_create(parent, vf_id);
> > +	if (IS_ERR(mdev)) {
> > +		ret =3D PTR_ERR(mdev);
> > +		goto err_rel;
> > +	}
> > +
> > +	ops =3D mdev->ops;
> > +	if (!ops || !ops->init || !ops->cleanup ||
> > +	    !ops->open || !ops->close ||
> > +	    !ops->save_state || !ops->load_state ||
> > +	    !ops->suspend || !ops->resume) {
> > +		ret =3D -EIO;
> > +		dev_err(&parent->dev, "Incomplete device migration ops
> > structure!");
> > +		goto err_destroy;
> > +	}
>=20
> If all these ops are a must why cant we move the check inside the
> qat_vfmig_create()?
> Or rather call them explicitly as suggested by Jason.

We can do it, but it might make sense to leave the check to the APIs' user
as some of these ops interfaces might be optional for other QAT variant dri=
ver
in future.

Thanks,
Xin