Received-SPF: pass (google.com: domain of linux-crypto-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 27 Sep 2021 13:06:27 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Leon Romanovsky <leonro@nvidia.com>
Cc:     Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "mgurtovoy@nvidia.com" <mgurtovoy@nvidia.com>,
        liulongfang <liulongfang@huawei.com>,
        "Zengtao (B)" <prime.zeng@hisilicon.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        "Wangzhou (B)" <wangzhou1@hisilicon.com>
Subject: Re: [PATCH v3 6/6] hisi_acc_vfio_pci: Add support for VFIO live
 migration
Message-ID: <20210927160627.GC964074@nvidia.com>
References: <20210915095037.1149-1-shameerali.kolothum.thodi@huawei.com>
 <20210915095037.1149-7-shameerali.kolothum.thodi@huawei.com>
 <20210915130742.GJ4065468@nvidia.com>
 <fe5d6659e28244da82b7028b403e11ae@huawei.com>
 <20210916135833.GB327412@nvidia.com>
 <a440256250c14182b9eefc77d5d399b8@huawei.com>
 <20210927150119.GB964074@nvidia.com>
 <YVHqlzyIX3099Gtz@unreal>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YVHqlzyIX3099Gtz@unreal>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sGOc7ydlgaqsIft1gPvRPkrFkXSzu0SW7RGWXwWIlZvbYqAWv5qdlNWPnuTs?=
 =?us-ascii?Q?GtBwkwKnnkPA+4Jhb0Y1JpiOJgBVbYr4wUv0SKbC4asUvIjDcvedQaA0S0a/?=
 =?us-ascii?Q?VzaA26cqv++4T8okMx62d3iLR6rGfofIR4/wET082BLKnOR59fffnRRsWjuL?=
 =?us-ascii?Q?joRnPWUOKj0DDT/VKGOXX8v3OMKf9UTwFVzr4IxR4jORrSzZcf4OefdeEYaU?=
 =?us-ascii?Q?0Ko0WOD9G5Qc33S0M1wvM96WYzPqJ5BExqwPFSzq3PqzlLnPkAa+5qyijXdT?=
 =?us-ascii?Q?zSx/6qNi+5UGAC42Y8Ovw13L/et8hRQr4YYsReWMCRkPOQVdeE17tVJXMp/4?=
 =?us-ascii?Q?Xksv14F0pAxBQDOvZhMozkhr8rBZGlf8zAi8g9MQnIiWl/K/B5G9xX8mLomE?=
 =?us-ascii?Q?/jR4SeBrprbif77eGATXj4u0QLgaco4OlqiqdGzeSrnark8udsnqwrgpJuZ0?=
 =?us-ascii?Q?X1AaItrLl/7v5zARAGxl3Hj9UBTmxjFGUrbLuCEwAYlvOgU9LRVM9AEN72jS?=
 =?us-ascii?Q?FR7VIMBb1AoO7xDSMwAVlx5nU5tey9Evll/tqjQN+aefn2r9XMj/ZnlXwN1Q?=
 =?us-ascii?Q?0eH/zrGfM47wt1PTcz27bPyPwePb6fu0vnUHhE6Zkpme4RMq5aiojFiJPi5G?=
 =?us-ascii?Q?qhy+9wZNIxWMv3vPfkH608OwY/0eOe2AywqwRdmGPKIv0+AmuaGrwxvCOj1Q?=
 =?us-ascii?Q?NagCfWvEpaPxO4xOfVU9vSONXwuxf+2ULb1btmQajZ+TzryKKgt8H/0LQzcA?=
 =?us-ascii?Q?Mm0cfAGiRjdxIAO6mPyCb49BYkcAoLKDPkrm+oR6ieXrm6Z9xb0QRBhcR99n?=
 =?us-ascii?Q?Rw0UDDvQgvvSQbI7o8o9+ljPR6idbjYG+9aqOSJA9lRqLlDBNn8VrdnprR0V?=
 =?us-ascii?Q?EwVcvMcDW9qpnPux4Q0E67wPtr8ldcSGiezR1T3C4AifzbqW7Jaca4IyHcOD?=
 =?us-ascii?Q?W2+EisWfYBGtp1HDMiE56alt/gtEc8NfvbRBfLETvWN/rYArxK9J1+6QYrMK?=
 =?us-ascii?Q?L0TYflBZ6Rh7myoe69BC2XHKjvBNImB/DTW9+qwIXijW8QqqhDd5MMPUqj15?=
 =?us-ascii?Q?wm+XQKBlmxNxTZjmj+X7OZvr50K3bfEgNKoZczgzN/i9EguomtWi9DHHvkjX?=
 =?us-ascii?Q?YO4+XSyyc/JDQI8AbptRtVG9DWcb1p1RhZmFYVQcL2iLt/mts2KQnH4U65aH?=
 =?us-ascii?Q?rwhD33RZp4KDCtBTPQeW7KcTmmuA+dr7fosWtgs44Z7N9e9VNk4C+A4oPw9X?=
 =?us-ascii?Q?dNVUVyJHU/bD/Osk0S6wZz+IXDUI+BEh2d338o1Q8QBHP35ALtIYpkt5EOfX?=
 =?us-ascii?Q?YLlZSdmma5iDIpeyEfrpon3/?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: eacff957-0cac-4240-9c82-08d981d0c361
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2021 16:06:28.2284
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VwuAab6Cplqi6v7PHyKCbSft3gALHrjQBRtgxXVbzDsD9P+7jnSCJpxL+7Xw8FpG
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5111
Precedence: bulk
List-ID: <linux-crypto.vger.kernel.org>
X-Mailing-List: linux-crypto@vger.kernel.org

On Mon, Sep 27, 2021 at 07:00:23PM +0300, Leon Romanovsky wrote:
> On Mon, Sep 27, 2021 at 12:01:19PM -0300, Jason Gunthorpe wrote:
> > On Mon, Sep 27, 2021 at 01:46:31PM +0000, Shameerali Kolothum Thodi wrote:
> > 
> > > > > > Nope, this is locked wrong and has no lifetime management.
> > > > >
> > > > > Ok. Holding the device_lock() sufficient here?
> > > > 
> > > > You can't hold a hisi_qm pointer with some kind of lifecycle
> > > > management of that pointer. device_lock/etc is necessary to call
> > > > pci_get_drvdata()
> > > 
> > > Since this migration driver only supports VF devices and the PF
> > > driver will not be removed until all the VF devices gets removed,
> > > is the locking necessary here?
> > 
> > Oh.. That is really busted up. pci_sriov_disable() is called under the
> > device_lock(pf) and obtains the device_lock(vf).
> 
> Yes, indirectly, but yes.
> 
> > 
> > This means a VF driver can never use the device_lock(pf), otherwise it
> > can deadlock itself if PF removal triggers VF removal.
> 
> VF can use pci_dev_trylock() on PF to prevent PF removal.

no, no here, the device_lock is used in too many places for a trylock
to be appropriate

> > 
> > But you can't access these members without using the device_lock(), as
> > there really are no safety guarentees..
> > 
> > The mlx5 patches have this same sketchy problem.
> > 
> > We may need a new special function 'pci_get_sriov_pf_devdata()' that
> > confirms the vf/pf relationship and explicitly interlocks with the
> > pci_sriov_enable/disable instead of using device_lock()
> > 
> > Leon, what do you think?
> 
> I see pci_dev_lock() and similar functions, they are easier to
> understand that specific pci_get_sriov_pf_devdata().

That is just a wrapper for device_lock - it doesnt help anything

The point is to all out a different locking regime that relies on the
sriov enable/disable removing the VF struct devices

Jason