Received-SPF: pass (google.com: domain of linux-crypto+bounces-3818-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 24 Apr 2024 11:17:16 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Zeng, Xin" <xin.zeng@intel.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	"herbert@gondor.apana.org.au" <herbert@gondor.apana.org.au>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	qat-linux <qat-linux@intel.com>, "Cao, Yahui" <yahui.cao@intel.com>
Subject: Re: [PATCH v6 1/1] vfio/qat: Add vfio_pci driver for Intel QAT
 SR-IOV VF devices
Message-ID: <20240424141716.GO941030@nvidia.com>
References: <20240417143141.1909824-1-xin.zeng@intel.com>
 <20240417143141.1909824-2-xin.zeng@intel.com>
 <20240418165434.1da52cf0.alex.williamson@redhat.com>
 <BN9PR11MB52767D5F7FF5D6498C974B388C0D2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240419141057.GG3050601@nvidia.com>
 <BN9PR11MB52768D9ADA48E18CB99BCB768C112@BN9PR11MB5276.namprd11.prod.outlook.com>
 <DM4PR11MB55029F7EACB6EB257DCF37BA88102@DM4PR11MB5502.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <DM4PR11MB55029F7EACB6EB257DCF37BA88102@DM4PR11MB5502.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?WX3CJob6p/yEGk+97C77HT3PdcflYuUQGe57NCYapj+k+3B+r6b0u3jAP+QP?=
 =?us-ascii?Q?IyynX/efAQ8pjNveC5ggT7TjKDRmVbMFGpZcTCzO2M820hy/HxIFmq4e1YL+?=
 =?us-ascii?Q?ToM81eXp0WEW0FCzsXNPXPGA4lCXvFcFTBT5ZshcFAH14bja8dzn9zoEQRuH?=
 =?us-ascii?Q?1lYin8ZQZJbxzx+anS+rYUFb3mNCy8cJRh0b/u119dRAyy1bQJ3BU1CZaCne?=
 =?us-ascii?Q?tgAHjHwoVGaI6ecS2MuYvuSZEIYVINP6oXz5o8r0PRjhtzT0Zv16oemkGOu4?=
 =?us-ascii?Q?qwHgW5zVZbURUiPWKSxeb34CLs7HX4XKav04sBvn0nhjQOxzQ7IEjqFMceGP?=
 =?us-ascii?Q?M/xvvhY5haOwxhWUdarNCH5MDSIkzKA+tmi4cGNl2pHIW+BFb9Xodd2/HBZz?=
 =?us-ascii?Q?dhLNR/mOvWrm8RhnhrmPZJd7dZG6wp8pzcWp9/Y00gssPQdIjTBwGHWRCv6D?=
 =?us-ascii?Q?PPxQ3Qc7+dVms6w/av41ETtTgU+zPajZffV1VmLMsi9TybkXpESdhCz80OYg?=
 =?us-ascii?Q?9tmQKFlEhY/JNkK34HmJDXEPYTYE2F15bQ0VmGaMU6E/dXPvZQNLSb6++iKL?=
 =?us-ascii?Q?hZYdtlh27nV+jncVIeLZgKIlSgCEYYdik0QXvcw6/pyddMC3FnUsSVEf0nJ4?=
 =?us-ascii?Q?iBvdW56Yu/iWCHnouvZGuRoKW+oNt9whm0JcESExeiV3/n8BriHVGkWKCYzI?=
 =?us-ascii?Q?TVGbOkN7Y7vh3C/blvQduRzq0wMWzBLxjYQ+KfEabSF9+8kKqN1EvjNJl+12?=
 =?us-ascii?Q?4dxFpyUX7N1cHlpRoBwJig32EWDbbN/PBiTDJ6ShtEH8i5eccGJD/LopLJ93?=
 =?us-ascii?Q?KX/RsNNdp7D/89k9IO+OZFW3iDbce1pdSumclfkzcN8A0JQ0ZrC/m15weFom?=
 =?us-ascii?Q?QOX4sBb24ypBWeJBzbotqptVNG6/+PybXCZ5q6gJGff4vV/vRwqtzajsa/NN?=
 =?us-ascii?Q?nhCFCcpPSIaJoU40BWMtV4+UaIAYGsipYSpp4QdZevqAAIR2nPXcxW8wxDJG?=
 =?us-ascii?Q?MAszgh881D0JYdUNwnZ0TPsoITBd9R8FMmQNxdK0iFTGPbaCKgVXHWWtTH8M?=
 =?us-ascii?Q?AunN1XbLyPD0iYrnfydcgmYARR+38VRLfAtjW4brEyJ8qNbpm5C4KHcXhNmh?=
 =?us-ascii?Q?Q2/LI6C3+Bhbq7LrublxvWfRZduR3TC3wvJY9u8A6klP4GZq3G0LU+J/KCK5?=
 =?us-ascii?Q?aDTiOLCIwqGp6puPpTVeabQJt3ufJml679TcmtGEeeZ/cNdLcBprsBdlQuI1?=
 =?us-ascii?Q?96Om0gVePEuuqV5bSi2pJk5eU+cPXO0P5RT/yqL5kGw4OVqlosLoAbDJ+d99?=
 =?us-ascii?Q?PTlaiR9tk03JvjXq8Samh2ySSUD+N88Fk5za6piiiP7QzuTLSejmbZFaZuHi?=
 =?us-ascii?Q?sdBYmyFQkjz7wbb1hh5qlFgImAPU7XkTRMox4Y0qmLOAdYE1vJ6NcKIGQo9b?=
 =?us-ascii?Q?SshOwqr5OpbG85TzuXuRygKaWEKJKI/jB4gE4eQax7qKCZ4oAEY61NveJ72K?=
 =?us-ascii?Q?7Cj2Vu5fNeJBlsvI3H4eNjKklns3hvHMeu+vekjT1QOiYVnJbb4wi9Txq+vh?=
 =?us-ascii?Q?4L+wimQaICjL5YxnaJFOINSBja+XOluc0DBVb/j4?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0cebf64b-17bf-45b2-aee1-08dc64693fd0
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 14:17:18.5995
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YvYLGA1pZWFLJKtM/hV/dkUjnbbWECrjEEnVrC6522ncyQNkBnJ5X1VzPqKjSpUV
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5872

On Wed, Apr 24, 2024 at 03:34:19AM +0000, Zeng, Xin wrote:

> > > Like if it halts all queues and keeps them halted, while still
> > > allowing queue head/tail pointer updats then it would be a fine
> > > implementation for P2P.
> > 
> > Yes that really depends. e.g. a queue accepting direct stores (MOVDIR64B)
> > for work submission may have problem if that store is simply abandoned
> > when the queue is disabled. ENQCMD is possibly OK as unaccepted store
> > will get a retry indicator to software so nothing is lost.
> > 
> > I'll let Xin confirm on the QAT implementation (for all device registers).
> > If it is like Jason's example then we should provide a clear comment
> > clarifying that doing suspend at RUNNING_P2P is safe for QAT as the
> > device MMIO interface still works according to the definition of RUNNING
> > and no request is lost (either from CPU or peer). There is nothing to stop
> > from RUNNING_P2P to STOP because the device doesn't execute any
> > request to further change the internal state other than MMIO.
> > 
> 
> When QAT VF is suspended, all its MMIO registers can still be operated
> correctly, jobs submitted through ring are queued while no jobs are
> processed by the device.  The MMIO states can be safely migrated
> to the target VF during stop-copy stage and restored correctly in the
> target VF. All queued jobs can be resumed then. 
> MOVDIR64B mentioned above is not supported by QAT solution,
> so it's fine to keep current implementation.
> If no objections, I'll append this paragraph of comment in the driver and
> post another version.

Yes that looks good to me.

Jason