Received-SPF: pass (google.com: domain of linux-crypto+bounces-2395-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Thu, 29 Feb 2024 10:10:23 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Xin Zeng <xin.zeng@intel.com>, herbert@gondor.apana.org.au,
	yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com,
	kevin.tian@intel.com, linux-crypto@vger.kernel.org,
	kvm@vger.kernel.org, qat-linux@intel.com,
	Yahui Cao <yahui.cao@intel.com>
Subject: Re: [PATCH v3 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF
 devices
Message-ID: <20240229141023.GF9179@nvidia.com>
References: <20240221155008.960369-1-xin.zeng@intel.com>
 <20240221155008.960369-11-xin.zeng@intel.com>
 <20240226115556.3f494157.alex.williamson@redhat.com>
 <20240226191220.GM13330@nvidia.com>
 <20240226124107.4317b3c3.alex.williamson@redhat.com>
 <20240226194952.GO13330@nvidia.com>
 <20240226152458.2b8a0f83.alex.williamson@redhat.com>
 <20240228120706.715bc385.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240228120706.715bc385.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?mLHo+UpBQpccpUJLEmKr/p0oAga5ClqWuliMYYkpf+nl+SMPucj7AcWcbKcR?=
 =?us-ascii?Q?1E3hxyX+N6cPHgLjc8mIZNlETtnk+AdaIf8QY5Z84ExY1DPzFqEH8Cttio8S?=
 =?us-ascii?Q?tQsRFrKMU3dnPOsVOWQfOAq8Wbx3cWXIDo+HYekLS5mkowqXc14bl4iJm92D?=
 =?us-ascii?Q?rG84eeOpAkCms/SSSP9O9xNlV3hYUZksJ+F9fJ+KNaGEGoPNnvFuOzp3dtUx?=
 =?us-ascii?Q?8uFjpgAOQSYevU1rMH4pWKC3eaN3BLv/o7SpTi7Dxp1to/Mev+PzB5tbOnOJ?=
 =?us-ascii?Q?InXrsNRzyk3RrnrgdugeOKh8l5mXBbYXWlzMEsudQtl+MBSuHzCGVGLDHGXe?=
 =?us-ascii?Q?W1I5a7QhSTK6ozBetFhN4111dFIIONr3iSJ3cRjJzCFWY9vshTL6V5sHdObr?=
 =?us-ascii?Q?r8xODUYgQrwokzljbceQ8s3nRLg8FHqWgkeXvc3DFJNk9acydsQVh6B4g6kt?=
 =?us-ascii?Q?/JIO3a0oW2b7oNNcSNA7ETohP9TCu/PvvEQnPczJ7dlCPR9HpQfPhEmHOPAD?=
 =?us-ascii?Q?AE7Lt77Z/Qb5xzBw7WslEVd0cIgtkEg7nJgMBO/n4wyRBqJWCzBXSEsep6Wi?=
 =?us-ascii?Q?yboJ3N1RkrApvyWJAfiSAEWH0IZkKRt1tz8SA5neLgrVTo/Eds4X2Jc7v1rp?=
 =?us-ascii?Q?rIODWQX2EdBQ49DeT7YYszw1AQkMDhpBJC6lpOATjXz/Fv/kEYVn8j/AyK4D?=
 =?us-ascii?Q?YHF+i1Oku71b6OpKLnQcIcmtnWSzqCuGX0dtCQd3WPYyoI6dP4rk07TYEpxh?=
 =?us-ascii?Q?UZd0q812iHIfLJxyjdNvTXgGJasNOZ10P1SWcgpOwBq0zgp5QT/Upi9nteEj?=
 =?us-ascii?Q?0Xzy4aFWhN6I3yAK7PkXGQLBjO33/x9rVeHXzRGphE/BTmQnkP/Jz6oHQrNr?=
 =?us-ascii?Q?hU8/hA9G4HqDUPNk0noMDQDL1ttvxNcCAyKfV2i81D7fxIezG3lgN8v4T9nY?=
 =?us-ascii?Q?dyAFqNszwVanJQUJqO5mKl0WxE8QHhz9vISWzZVcvEk+JkTq4/oTpzNlpPlA?=
 =?us-ascii?Q?gJiST4fjDHfeU3bC2GLXKwDoy3Yfb3Vgl+hoe1Rzfz94xNKc+5APJGfTdTQP?=
 =?us-ascii?Q?7nGUwUt6z1bwFSYGLiz3y4i+wnXi/3+Yj1pOBIbnVhSA1m9Z8HQ/yL4yS+Dc?=
 =?us-ascii?Q?eZMpWudfJD7Kdiho975kWpFWWLViVb/0Gt5SYUMZq1XQ05VvkEFyCQb2RP5p?=
 =?us-ascii?Q?WKYFJeWH4bgJqBcvl1eZX/s65AxL5mOzbsRPqIFe42MvtAshEO+hdkVy7qzS?=
 =?us-ascii?Q?RzMDZN28YEJxK+ddnCLBp/0UExFG6p9KhGA9eKQl7qr5cK75clL0Uc5mgRqS?=
 =?us-ascii?Q?3+oKggnOnuHVX6pJoppkxf+DsQoVMSNnvXDtOmO/2QdjX6ycp2XxfOB8ptxn?=
 =?us-ascii?Q?qgJagru6aeiKalcuV+rBFoi2RjLr6x9RwoSYJMFmE+cs6lLAIoc1cfu/PNsq?=
 =?us-ascii?Q?AVftmU0Q6tHScsqYGFUsFXbsbjUMm0euxpERkCoEIbItTKgJLvlVdxQfkulK?=
 =?us-ascii?Q?OZvOBFwIt5x/GAKmfd9ozZ9yPhYiwDVcDjHGpsfNG4OXp6lpHIe9Hyh1a6EX?=
 =?us-ascii?Q?dFgJ0sJZ8oEJEWwZMmGVSSQb+kOwLGeEoV6BtKc1?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 085f9652-16ed-453c-c921-08dc39302c48
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Feb 2024 14:10:24.5044
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fUQt7QNETO7EXUVonig/chnIqdzERpnT6QpbuEMArzkhLgemhpPiF1lnVdN/BHEt
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8056

On Wed, Feb 28, 2024 at 12:07:06PM -0700, Alex Williamson wrote:
> On Mon, 26 Feb 2024 15:24:58 -0700
> Alex Williamson <alex.williamson@redhat.com> wrote:
> 
> > On Mon, 26 Feb 2024 15:49:52 -0400
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> > 
> > > On Mon, Feb 26, 2024 at 12:41:07PM -0700, Alex Williamson wrote:  
> > > > On Mon, 26 Feb 2024 15:12:20 -0400
> > > > libvirt recently implemented support for managed="yes" with variant
> > > > drivers where it will find the best "vfio_pci" driver for a device
> > > > using an algorithm like Max suggested, but in practice it's not clear
> > > > how useful that will be considering devices likes CX7 require
> > > > configuration before binding to the variant driver.  libvirt has no
> > > > hooks to specify or perform configuration at that point.    
> > > 
> > > I don't think this is fully accurate (or at least not what was
> > > intended), the VFIO device can be configured any time up until the VM
> > > mlx5 driver reaches the device startup.
> > > 
> > > Is something preventing this? Did we accidentally cache the migratable
> > > flag in vfio or something??  
> > 
> > I don't think so, I think this was just the policy we had decided
> > relative to profiling VFs when they're created rather than providing a
> > means to do it though a common vfio variant driver interface[1].
>
> Turns out that yes, migration support needs to be established at probe
> time.  vfio_pci_core_register_device() expects migration_flags,
> mig_ops, and log_ops to all be established by this point, which for
> mlx5-vfio-pci occurs when the .init function calls
> mlx5vf_cmd_set_migratable().

This is unfortunate, we should look at trying to accomodate this,
IMHO. Yishai?

> That also makes me wonder what happens if migration support is disabled
> via devlink after binding the VF to mlx5-vfio-pci.  Arguably this could
> be considered user error, but what's the failure mode and support
> implication?  Thanks,

I think tThe FW will start failing the migration commands.

Jason