Received-SPF: pass (google.com: domain of linux-kernel+bounces-64828-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.161 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C
Date: Wed, 14 Feb 2024 09:16:18 +0200
From: Leon Romanovsky <leonro@nvidia.com>
To: Bjorn Helgaas <helgaas@kernel.org>
CC: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>,
	Jim Harris <jim.harris@samsung.com>, Bjorn Helgaas <bhelgaas@google.com>,
	<linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>, Jason Gunthorpe
	<jgg@nvidia.com>, Alex Williamson <alex.williamson@redhat.com>, Pierre
 =?iso-8859-1?Q?Cr=E9gut?= <pierre.cregut@orange.com>
Subject: Re: [PATCH v2 1/2] PCI/IOV: Revert "PCI/IOV: Serialize sysfs
 sriov_numvfs reads vs writes"
Message-ID: <20240214071618.GE52640@unreal>
References: <20240213174602.GD52640@unreal>
 <20240213194556.GA1219770@bhelgaas>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240213194556.GA1219770@bhelgaas>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2024 07:16:32.1354
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 70244199-0eb7-469f-4898-08dc2d2cdf00
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource:
	DS1PEPF0001709C.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4562

On Tue, Feb 13, 2024 at 01:45:56PM -0600, Bjorn Helgaas wrote:
> On Tue, Feb 13, 2024 at 07:46:02PM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 13, 2024 at 09:59:54AM -0600, Bjorn Helgaas wrote:
> > ...
> 
> > > I guess that means that if we apply this revert, the problem Pierre
> > > reported will return.  Obviously the deadlock is more important than
> > > the inconsistency Pierre observed, but from the user's point of view
> > > this will look like a regression.
> > > 
> > > Maybe listening to netlink and then looking at sysfs isn't the
> > > "correct" way to do this, but I don't want to just casually break
> > > existing user code.  If we do contemplate doing the revert, at the
> > > very least we should include specific details about what the user code
> > > *should* do instead, at the level of the actual commands to use
> > > instead of "ip monitor dev; cat ${path}/device/sriov_numvfs".
> > 
> > udevadm monitor will do the trick.
> > 
> > Another possible solution is to refactor the code to make sure that
> > .probe on VFs happens only after sriov_numvfs is updated.
> 
> I like the idea of refactoring it so as to preserve the existing
> ordering while also fixing the deadlock.

I think something like this will be enough (not tested). It will et the number of VFs
before we make VFs visible to probe:

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index aaa33e8dc4c9..0cdfaae80594 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -679,12 +679,14 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 	msleep(100);
 	pci_cfg_access_unlock(dev);
 
+	iov->num_VFs = nr_virtfn;
 	rc = sriov_add_vfs(dev, initial);
-	if (rc)
+	if (rc) {
+		iov->num_VFs = 0;
 		goto err_pcibios;
+	}
 
 	kobject_uevent(&dev->dev.kobj, KOBJ_CHANGE);
-	iov->num_VFs = nr_virtfn;
 
 	return 0;