Received-SPF: pass (google.com: domain of linux-crypto-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.32 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.32; helo=mail.nvidia.com;
Date:   Mon, 27 Sep 2021 19:00:23 +0300
From:   Leon Romanovsky <leonro@nvidia.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "mgurtovoy@nvidia.com" <mgurtovoy@nvidia.com>,
        liulongfang <liulongfang@huawei.com>,
        "Zengtao (B)" <prime.zeng@hisilicon.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        "Wangzhou (B)" <wangzhou1@hisilicon.com>
Subject: Re: [PATCH v3 6/6] hisi_acc_vfio_pci: Add support for VFIO live
 migration
Message-ID: <YVHqlzyIX3099Gtz@unreal>
References: <20210915095037.1149-1-shameerali.kolothum.thodi@huawei.com>
 <20210915095037.1149-7-shameerali.kolothum.thodi@huawei.com>
 <20210915130742.GJ4065468@nvidia.com>
 <fe5d6659e28244da82b7028b403e11ae@huawei.com>
 <20210916135833.GB327412@nvidia.com>
 <a440256250c14182b9eefc77d5d399b8@huawei.com>
 <20210927150119.GB964074@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20210927150119.GB964074@nvidia.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2021 16:00:30.7883
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: daa283ea-ac40-412b-9bc8-08d981cfee7c
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT017.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5245
Precedence: bulk
List-ID: <linux-crypto.vger.kernel.org>
X-Mailing-List: linux-crypto@vger.kernel.org

On Mon, Sep 27, 2021 at 12:01:19PM -0300, Jason Gunthorpe wrote:
> On Mon, Sep 27, 2021 at 01:46:31PM +0000, Shameerali Kolothum Thodi wrote:
> 
> > > > > Nope, this is locked wrong and has no lifetime management.
> > > >
> > > > Ok. Holding the device_lock() sufficient here?
> > > 
> > > You can't hold a hisi_qm pointer with some kind of lifecycle
> > > management of that pointer. device_lock/etc is necessary to call
> > > pci_get_drvdata()
> > 
> > Since this migration driver only supports VF devices and the PF
> > driver will not be removed until all the VF devices gets removed,
> > is the locking necessary here?
> 
> Oh.. That is really busted up. pci_sriov_disable() is called under the
> device_lock(pf) and obtains the device_lock(vf).

Yes, indirectly, but yes.

> 
> This means a VF driver can never use the device_lock(pf), otherwise it
> can deadlock itself if PF removal triggers VF removal.

VF can use pci_dev_trylock() on PF to prevent PF removal.

> 
> But you can't access these members without using the device_lock(), as
> there really are no safety guarentees..
> 
> The mlx5 patches have this same sketchy problem.
> 
> We may need a new special function 'pci_get_sriov_pf_devdata()' that
> confirms the vf/pf relationship and explicitly interlocks with the
> pci_sriov_enable/disable instead of using device_lock()
> 
> Leon, what do you think?

I see pci_dev_lock() and similar functions, they are easier to
understand that specific pci_get_sriov_pf_devdata().

Thanks

> 
> Jason