Received-SPF: pass (google.com: domain of linux-crypto+bounces-3947-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: "Zeng, Xin" <xin.zeng@intel.com>
To: liulongfang <liulongfang@huawei.com>, "herbert@gondor.apana.org.au"
	<herbert@gondor.apana.org.au>, "alex.williamson@redhat.com"
	<alex.williamson@redhat.com>, "jgg@nvidia.com" <jgg@nvidia.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "Tian, Kevin" <kevin.tian@intel.com>
CC: "linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, qat-linux <qat-linux@intel.com>
Subject: RE: [PATCH v5 09/10] crypto: qat - implement interface for live
 migration
Thread-Topic: [PATCH v5 09/10] crypto: qat - implement interface for live
 migration
Thread-Index: AQHab8+8zwh8J6q/SU+vnDaXrayAibGAeCeAgAAv1bA=
Date: Tue, 30 Apr 2024 09:49:40 +0000
Message-ID: <DM4PR11MB55026F0FF713BCB3317F5AC7881A2@DM4PR11MB5502.namprd11.prod.outlook.com>
References: <20240306135855.4123535-1-xin.zeng@intel.com>
 <20240306135855.4123535-10-xin.zeng@intel.com>
 <7cedf6ec-3c8a-b6d8-d5fc-778554c011c2@huawei.com>
In-Reply-To: <7cedf6ec-3c8a-b6d8-d5fc-778554c011c2@huawei.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?3bnwQPzz3jtbCcDewNpBjtL+LFm9GpJLzydtGSo+T+xV6Fp50xc5++/xQHjk?=
 =?us-ascii?Q?ZyoPvgEYzZGF0zyuXpafbAx9vV/wukE+M1A6ahrxFfnfX4AgCmBYr2c4Ek39?=
 =?us-ascii?Q?vy42u1Yh5LjsyZLA4BPkvpay+6q8Ys9fOM6gmXo3nSA3l/0ka7FM9zsk7qSh?=
 =?us-ascii?Q?qH5p+N4jkNbOA7wXnaEGi9759fORviR53DXpZPtvL+8ct5lLmmO+lUAsFXTy?=
 =?us-ascii?Q?diC2Q+NG75zfAbqQ88r2tc4aW1Jcljthd1M5PfVrMsb5xSfTXfDIsChQJkzG?=
 =?us-ascii?Q?LGMKq9UBQeEVwge2B96xzKksqjoM+X5nLQY0mjAqakLalRIRiNR70gkKh5Td?=
 =?us-ascii?Q?uodSuPYF12JdlEb7wlvgCoTstTn3m4l3ra/i+9GWOQNw2cwTOPypKdQR0+Ki?=
 =?us-ascii?Q?gmZk7UV1ekCWLlCDTJW9+4J/fo2l2GTL0c1huqbavccn3DHpJTftytTLonRd?=
 =?us-ascii?Q?4wyw7M6Ub3zs4IBu0WqFY0CA4oMQaMGjm7xBHdE+SoN9ZNf0UOhmbYGW7/mS?=
 =?us-ascii?Q?EGNLPEkHaTeO0osnFGhUPNYttmQwaRZm460fSWr2zgbRM7B+ChKZbXbpF7qO?=
 =?us-ascii?Q?Tol2NV052d3gg1tu6gShBcwtSGmLzvGXWoJYTdyT7UCw217aosfdDsxd6WXp?=
 =?us-ascii?Q?MdmJaeOd8wQEjpf0c74llCdeGtyRnKQRYgskcJ0tjE2isQ4NYvuR4jpngvpL?=
 =?us-ascii?Q?adwTmEaS6fO3HhnG2SER3fI3QhuqZQpqJQr14LS/RLE9nrqt1cxeXURFCgkG?=
 =?us-ascii?Q?r2vXNEufRRxumSQFH6lmmqGM/lgKCPbKJUV4ea8pFO7Tnknz5YqYdJz+k52I?=
 =?us-ascii?Q?7qOU7/BOVkp147phHgIvBUWwuVezEuHmc3Ku0duKmHNm+b/cCsxeNtsz6C91?=
 =?us-ascii?Q?9chZZoFdYWrZ178P+Es+bU81t4OfFVqNhcPZdk4chAtYHo7cKV7VHKjHYh8Q?=
 =?us-ascii?Q?pR9e+5EEtZlmnyWHp/nK0hLwW4Xnv2lt1/PKHlz3O0ZJxAJ2ngXEsCmQ+KDC?=
 =?us-ascii?Q?5G0a6+x/4b5NN1Finyni8XRd3oOplyTSAs7cBR7Uatjr1fQLCqi0Ey6gCZBJ?=
 =?us-ascii?Q?g/Blo/PKrn+pUZfDVAkQbr43b3yRD6HGSXR+7XQbfUH1mYUFNK18aeNQGdrh?=
 =?us-ascii?Q?d9QcLOLrgLwB+cLVGq5OHjbBYad7zdFqFMGY92WRLrySvS3HGYH8V6MQuf2I?=
 =?us-ascii?Q?QMUqEhDv16FdHmBCLqnPo0+7Uzh7zLC9uef7AumCJSYyY2znSLS/j3dpCmpG?=
 =?us-ascii?Q?LDSJdYZKi1Z5YH0cZkA2gTq0jqyrJ8r46JerQ9tjqPw+s2cSD5tertwRTrNr?=
 =?us-ascii?Q?buUbRFiCUhH3AZWghiZ1UT1NKsuCcp/W47tiGBFVXjU7yhvVHKJPytV7T5kt?=
 =?us-ascii?Q?/TITL8VbAfpGGwwU9pgLx9s+y01YU30Og7aEx8MO4qKv35F6l/c4j+aSOqrc?=
 =?us-ascii?Q?jmoZRKi2K0skfSRUxFZU3uG7TrbJfjbix4UhZ8BZYmExgUB76VekPe4VjXSE?=
 =?us-ascii?Q?ww1mB1+A5KAi6rigwW2ttBtWdw3JmHXNAySBFus+X6qZXs3ultVgSFWXJ1ot?=
 =?us-ascii?Q?ZRZPfir0OGQOjMgLMO2Q8//jBS5M3yPc8TH6unwF?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-crypto@vger.kernel.org
List-Id: <linux-crypto.vger.kernel.org>
List-Subscribe: <mailto:linux-crypto+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-crypto+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5502.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 446819a1-42c8-4e62-b88f-08dc68fadb41
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2024 09:49:40.9149
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: hwnnIQ+P/frjlCWfxssDTHsUlKDo4Qzm3hGe8Mq/D2YZ/Jzw14Nao5qclKArSyErUfox/b0SkOxhljKlflvBMA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4825
X-OriginatorOrg: intel.com

On Tuesday, April 30, 2024 11:10 AM, liulongfang <liulongfang@huawei.com> w=
rote:
> To: Zeng, Xin <xin.zeng@intel.com>; herbert@gondor.apana.org.au;
> alex.williamson@redhat.com; jgg@nvidia.com; yishaih@nvidia.com;
> shameerali.kolothum.thodi@huawei.com; Tian, Kevin <kevin.tian@intel.com>
> Cc: linux-crypto@vger.kernel.org; kvm@vger.kernel.org; qat-linux <qat-
> linux@intel.com>
> Subject: Re: [PATCH v5 09/10] crypto: qat - implement interface for live
> migration
>=20
> On 2024/3/6 21:58, Xin Zeng wrote:
> > Add logic to implement the interface for live migration defined in
> > qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions
> > (VFs).
> >
> > This introduces a migration data manager which is used to handle the
> > device state during migration. The manager ensures that the device stat=
e
> > is stored in a format that can be restored in the destination node.
> >
> > The VF state is organized into a hierarchical structure that includes a
> > preamble, a general state section, a MISC bar section and an ETR bar
> > section. The latter contains the state of the 4 ring pairs contained on
> > a VF. Here is a graphical representation of the state:
> >
> >     preamble | general state section | leaf state
> >              | MISC bar state section| leaf state
> >              | ETR bar state section | bank0 state section | leaf state
> >                                      | bank1 state section | leaf state
> >                                      | bank2 state section | leaf state
> >                                      | bank3 state section | leaf state
> >
> > In addition to the implementation of the qat_migdev_ops interface and
> > the state manager framework, add a mutex in pfvf to avoid pf2vf message=
s
> > during migration.
> >
> > Signed-off-by: Xin Zeng <xin.zeng@intel.com>
> > Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
> > ---
> >  .../intel/qat/qat_420xx/adf_420xx_hw_data.c   |    2 +
> >  .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |    2 +
> >  drivers/crypto/intel/qat/qat_common/Makefile  |    2 +
> >  .../intel/qat/qat_common/adf_accel_devices.h  |    6 +
> >  .../intel/qat/qat_common/adf_gen4_hw_data.h   |   10 +
> >  .../intel/qat/qat_common/adf_gen4_vf_mig.c    | 1010 +++++++++++++++++
> >  .../intel/qat/qat_common/adf_mstate_mgr.c     |  318 ++++++
> >  .../intel/qat/qat_common/adf_mstate_mgr.h     |   89 ++
> >  .../crypto/intel/qat/qat_common/adf_sriov.c   |    7 +-
> >  9 files changed, 1445 insertions(+), 1 deletion(-)
> >  create mode 100644
> drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c
> >  create mode 100644
> drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c
> >  create mode 100644
> drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h
> >
> > diff --git a/drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
> b/drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
> > index 9ccbf5998d5c..d255cb3ebd9c 100644
> > --- a/drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
> > +++ b/drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
> > @@ -17,6 +17,7 @@
> > +void adf_gen4_init_vf_mig_ops(struct qat_migdev_ops *vfmig_ops)
> > +{
> > +	vfmig_ops->init =3D adf_gen4_vfmig_init_device;
> > +	vfmig_ops->cleanup =3D adf_gen4_vfmig_cleanup_device;
> > +	vfmig_ops->reset =3D adf_gen4_vfmig_reset_device;
> > +	vfmig_ops->open =3D adf_gen4_vfmig_open_device;
> > +	vfmig_ops->close =3D adf_gen4_vfmig_close_device;
> > +	vfmig_ops->suspend =3D adf_gen4_vfmig_suspend_device;
> > +	vfmig_ops->resume =3D adf_gen4_vfmig_resume_device;
> > +	vfmig_ops->save_state =3D adf_gen4_vfmig_save_state;
> > +	vfmig_ops->load_state =3D adf_gen4_vfmig_load_state;
> > +	vfmig_ops->load_setup =3D adf_gen4_vfmig_load_setup;
> > +	vfmig_ops->save_setup =3D adf_gen4_vfmig_save_setup;
> > +}
> > +EXPORT_SYMBOL_GPL(adf_gen4_init_vf_mig_ops);
>=20
> This GEN4 device supports live migration functionality.
> The above part of the code supports the live migration function and
> has nothing to do with crypto.
>=20
> Therefore, these should be moved to the vfio/pci/qat directory.
>=20

Thanks for the suggestion, but
1. The migration operations of QAT VF rely on QAT PF driver sitting in
crypto tree to handle. Some of the states can only be accessed from PF by
PF driver. For each generation of PF, we will have a PF driver. It is obvio=
us
more clear and nature to make these operations part QAT PF driver rather
than the variant VF driver.=20
2. The interfaces are defined clear enough to understand the dependency
of the variant migration VF driver to QAT PF driver.
3. A device driver sitting in crypto tree usually not only provides crypto =
stuff, but
also provide helpers to support other functionalities such as the non vfio
use space process access logic, it does make sense to provide migration hel=
pers in
device driver as well.
Those why I prefer to put these helpers into QAT PF driver.

Thanks